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ABSTRACT 


Littoral anti-submarine warfare (ASW) operations generally focus on deterring and 
eliminating enemy diesel-electric submarines from transit routes and protecting high 
value units (HVUs), such as amphibious warfare ships and logistics ships. In view of the 
ASW challenges in the littorals, it is critical to establish and maintain a highly effective 
ASW capability. The ASW techniques that we use today are mostly effective, but it is 
important to explore new technologies and techniques—such as potential unmanned 
surface vehicle (USV) solutions. This study uses an agent-based simulation platform 
known as Map Aware Non-Unifonn Automata (MANA) to model the ASW effectiveness 
of USVs with the goal of protecting a HVU. The effectiveness of an ASW screen 
formation is measured by the proportion of successful classifications. The results are 
analyzed using comparison methods, stepwise linear regression, and regression trees. It is 
found from nearly 390,000 simulated ASW missions that when helicopters are replaced 
with USVs, which have the same sensor type and capability, USVs can provide the same 
classification effectiveness in an ASW screen formation. The analysis also shows that the 
most significant characteristic of USVs is the classification range of their dipping sonar. 
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EXECUTIVE SUMMARY 


The Turkish naval fleet conducts operations in its littoral waters to ensure free access to 
international waters and to deter any threat to the sea lines of communications (SLOCs). 
Thus, antisubmarine warfare (ASW) operations in Turkish littoral waters generally focus 
on deterring and eliminating enemy diesel-electric submarines from transit routes and 
protecting naval assets and high value units (HVUs), such as amphibious and logistics 
ships. These operations enable naval forces to conduct more successful force protection 
and sealift operations and keep the SLOCs open and secure. 

Diesel-electric submarines are very quiet and stealthy—and pose a great threat to 
Turkey’s and allied forces’ SLOCs. With the increasing emphasis on littoral ASW, we 
should investigate complementary abilities to address and eliminate diesel-electric 
submarines with conventional forces. Technological enhancements bring us new 
capabilities to fight against stealthy underwater threats. Unmanned surface vehicles 
(US Vs) have the potential to enhance the current littoral ASW capabilities. US Vs have 
been used in naval operations since World War II, but recently these vehicles are gaining 
more interest from modem navies with their increased operational capabilities. 

Effective employment and the correct tactical use of USVs may offer a great force 
multiplier. This can bring operational success, reduced risk and casualties to manned 
platforms, and improved operational effectiveness. Based on the discussion above, this 
thesis examines the effectiveness of unmanned surface vehicles in anti-submarine warfare 
with the goal of protecting a high value unit. 

This study uses an agent-based simulation platform known as Map Aware Non- 
Uniform Automata (MANA) to model the ASW effectiveness of USVs while considering 
their advantage of long on-station time and disadvantage of low speed (as compared to 
helicopters). A generic scenario is created to allow us to experiment with potential USV 
capabilities in ASW missions. The modeling first focuses on building an existing ASW 
screening scenario in MANA. In this scenario, two frigates with hull-mounted active 
sonars are positioned on the inner ASW screen and two ASW helicopters with active 



dipping sonars are positioned on the outer ASW screen to protect an HVU from 
submarine attacks. This baseline scenario provides a standardized benchmark on current 
ASW performance. The battlefield characteristics and the overall representation of the 
baseline scenario are shown in the figure below. In the first alternative scenario, USVs 
are included in our model instead of helicopters. In doing so, USVs maintain a protective 
ASW barrier in front of the surface group. This model provides us some insights about 
USVs as to whether they can improve the effectiveness of ASW capabilities. Also, the 
model explores the overall effectiveness of ASW screening when USVs are employed 
with ASW helicopters. The same conditions are also explored for three frigate scenarios. 
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The battlefield characteristics and the overall representation of the baseline 
scenario (not drawn to scale). 


After modeling the scenarios in MANA, over 390,000 simulated ASW screening 
missions are executed. In designing our experiment, we apply a nearly orthogonal Latin 
hypercube (NOLH) design which provides good space-filling and statistical properties. 
We use the experimental design to vary controllable and uncontrollable factors and 

xviii 
























examine how they affect the ability to detect and classify a diesel-electric submarine 
attempting to attack an HVU. 

A comparison analysis is conducted among the scenarios with different numbers 
and varieties of platforms employed in an ASW screen formation. With side-by-side box 
plots and one-way analysis of the means by scenarios, it was found that when the 
helicopters are replaced with USVs, which have the same sensor type and capability, 
US Vs can provide the same classification effectiveness in an ASW screen formation. The 
operating range of the USVs is considerably shorter than the operating range of the 
helicopters because of the autonomy requirements of USVs. Therefore, USVs are 
employed in the intermediate screen while the helicopters are employed on the outer 
screen. This gives the helicopters a great advantage against USVs because the helicopters 
can extend the reach of the frigates to the farthest point in the ASW screen and provide 
an early detection and classification of the diesel-electric submarine. 

The proportion of successful classification is used to measure the effectiveness of 
ASW screen formation in a regression model. Based on this measure of effectiveness 
(MOE), the most significant characteristic of USVs is the classification range of their 
dipping sonar. In ASW, the classification range may depend on underwater conditions, 
background noise in the ocean, and sonar capability. The sonar parameters are mostly 
controllable because the selection of the sonar type and capability can be determined 
during the design process. But, the effectiveness of sonar is limited by environmental 
conditions. On the other hand, the speed is viewed as an insignificant characteristic of 
USVs in the model over the ranges explored. With this in mind, it is important that USVs 
self-deploy to the intennediate screen ahead of the HVU with sufficient time and 
endurance to satisfy their station-keeping requirements. 

Many decision and noise factors have a highly significant effect on the outcome 
in our protective ASW scenario. The sonar parameters of the frigates are especially 
significant in the model. The frigate sonar classification range has the greatest influence 
on ASW mission success. The number of frigates is another significant factor that affects 
the outcome. Employing one more frigate in the screening formation, along with its 
assets, significantly increases the probability of detecting hostile submarines. Among the 



noise factors, the stealthiness of the diesel-electric submarine plays an important role in 
the model, as expected, since it is a well- kn own crucial factor in littoral ASW operations. 
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I. 


INTRODUCTION 


“Peace at home, peace in the world. ” 

- Mustafa Kemal Ataturk 


Since the end of the Cold War, the threat environment has shifted from open seas 
to the brown waters, with a greater emphasis on expeditionary operations, power 
projection, and force protection in littoral waters. One of the greatest military challenges 
of today is modern diesel-electric submarines operating in noisy and cluttered littoral 
environments. Diesel-electric submarines are very quiet and stealthy—and pose a great 
threat to Turkey’s and allied forces’ sea lines of communications (SLOCs). With the 
increasing emphasis on littoral antisubmarine warfare (ASW), we should investigate 
complementary abilities to address and eliminate diesel-electric submarines with 
conventional forces. 

Technological enhancements bring us new capabilities to fight against stealthy 
underwater threats. Unmanned surface vehicles (USVs; see Figure 1) have the potential 
to enhance the current littoral ASW capabilities and reduce the risk to manned 
platforms [1]. USVs have been used in naval operations since World War II, but recently 
these vehicles are gaining more interest from modem navies with their increased 
operational capabilities [2]. 
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Figure 1. Unmanned surface vehicle (image from Textron Systems, 

http://www.textronsystems.com). 


A. OVERVIEW 

Over the past two decades, the littoral waters have gained great importance. In 
December 1991, as the world watched in great surprise, the fall of the Soviet Union put 
an end to the Cold War. The post-Cold War era has had a great effect on both political 
and military activities. This era raised the possibility of unpredictable regional wars, 
tensions, and conflicts, especially in the Middle East, Southwest Asia, Northern Africa, 
Western Pacific, and Eastern Europe. Today, it seems that in the case of possible 
conventional combat, naval activities will likely take place in littoral waters [3]. 

These naval activities include force protection, surveillance, littoral ASW, mine¬ 
hunting, mine-clearing, and support for amphibious operations. In the littoral battlespace, 
naval forces may encounter some threats from potential enemies that are different from 
those in open seas in the form of quiet diesel-electric submarines (see Figure 2). 
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Figure 2. A diesel-electric submarine (image from Jane’s Fighting Ships, 

https://janes.ihs.com). 


This unique platform is considered the deadliest threat in littoral waters because it 
can shut down its diesel engines and run on a battery charge when submerged, resulting 
in almost zero noise, and sail undetected for a long period of time. Moreover, high noise 
and poor sound propagation conditions in the littoral waters give the diesel submarine an 
even greater advantage. It can stay extremely quiet and submerged for up to one week. 
Many countries around the world operate modern diesel-electric submarines because they 
are relatively inexpensive and have greater effectiveness in littoral waters. Some common 
classes of modern diesel-electric submarines include Type 209, Type 212, Kilo-class, 
Dolphin-class, Scorpene, and Soryu [4]. Modem diesel-electric submarines can be used 
for many purposes, such as threatening vital shipping lanes and attacking high value units 
(HVUs) [5]. 

The main role of the Turkish Navy is to provide security for shipping lanes and 
protect Turkey’s rights and interests in its littoral waters, namely in the Aegean, Eastern 
Mediterranean, and the Black Sea (see Figure 3) [6]. 
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Figure 3. Turkey’s surrounding seas: The Black Sea, the Aegean Sea, and the 
Mediterranean Sea (image from The Encyclopedia of Earth, 
http://www.eoeearth.org). 

The Turkish naval fleet conducts operations in its littoral waters to ensure free 
access to international waters and to deter any threat to SLOCs. Thus, ASW operations in 
Turkish littoral waters generally focus on deterring and eliminating enemy diesel-electric 
submarines from transit routes and protecting naval assets and high value units (HVUs), 
such as amphibious and logistics ships. These operations enable naval forces to conduct 
more successful force protection and sealift operations and keep the SLOCs open and 
secure. 

Detecting a diesel-electric submarine is challenging and requires a variety of 
different platforms and sensors. Each platform has its own ASW capabilities and can be 
employed in various anti-submarine operations. To improve ASW effectiveness, these 
platforms and their sensors support each other [5]. Due to the operational challenges and 
importance of littoral waters, it is critical to establish and maintain a highly effective 
ASW capability [7]. Convoy or FIVU protection usually focuses on defensive ASW and 
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requires a detailed organization of escorting assets. In order to protect HVUs against 
possible submarine attacks, the Navy can employ surface warships, aircraft, helicopters, 
and unmanned underwater and surface vehicles (UUVs and USVs) equipped with active 
or passive sonar. These ASW assets are deployed to patrol certain areas relative to the 
HVU’s position [8], Each type of operation requires a certain number of ASW units, 
manpower, time, and money. 

The ASW techniques that we use today are mostly effective, but it is important to 
develop complementary skills, improve today’s technology, and explore new systems, 
such as unmanned solutions. This can increase the effectiveness of ASW capabilities in 
deterring and eliminating enemy submarines and protecting friendly forces. Given 
today’s increasing diesel-electric submarine threat from our enemies, it is important that 
the Navy has the capability of operating USVs in naval operations. Employing USVs in 
ASW operations has the potential to improve the efforts of existing ASW assets. 
Effective employment and the correct tactical use of USVs may offer a great force 
multiplier. This can bring us operational success, reduced risk and casualties to manned 
platforms, and improved operational effectiveness [1]. 

Based on the discussion above, this thesis examines the effectiveness of 
unmanned surface vehicles in anti-submarine warfare with the goal of protecting an 
HVU. 

B. RESEARCH QUESTIONS 

The research is guided by the following questions: 

1. Can USVs give the same effectiveness as ASW helicopters against diesel- 
electric submarines ahead of naval convoys or HVUs? 

2. What are the main advantages and disadvantages of employing USVs in 
an ASW screen fonnation? 

3. Which characteristics of USVs are the most significant in ASW? 

4. How do changes in decision parameters affect the probability of 
classifying a diesel-electric submarine? 

5. What strengths and drawbacks does the simulation software Map Aware 
Non-Uniform Automata (MANA) have for modelling ASW scenarios? 
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C. SCOPE AND METHODOLOGY 

This thesis explores how USVs can complement and extend existing ASW 
effectiveness in detecting and classifying diesel-electric submarines. This study also 
addresses many controllable and uncontrollable factors related to ASW to see which 
factors have the greatest effect on an ASW screen’s classification rate. Results will help 
decision-makers understand how USVs can be employed in an ASW screen fonnation. 

This thesis uses an agent-based simulation platform called MANA to model the 
ASW effectiveness of USVs while considering their advantage of long on-station time 
and disadvantage of low speed (relative to helicopters). Agent-based simulation is a 
technique in which we virtually construct multiple autonomous entities that make their 
own decisions and behave stochastically in their local environments [9]. 

The modeling first focuses on building an existing ASW screening scenario in 
MANA. In this scenario, two frigates with hull-mounted active sonars are positioned on 
the inner ASW screen and two ASW helicopters with active dipping sonars are 
positioned on the outer ASW screen to protect an HVU from submarine attacks. This 
baseline scenario provides us a standardized benchmark. In the first alternative scenario, 
USVs are included in our model instead of helicopters. In doing so, USVs will maintain a 
protective ASW barrier in front of the surface group. This model provides us some 
insights about USVs as to whether they can improve the effectiveness of ASW 
capabilities. Also, we explore the overall effectiveness of ASW screening when USVs are 
employed with ASW helicopters. The same conditions are also explored for three frigate 
scenarios. 

After modeling the scenarios in MANA, nearly 390,000 simulated ASW missions 
are executed. In designing our experiment, we apply a nearly orthogonal Latin hypercube 
(NOLH) design which provides good space-filling and statistical properties [10]. We use 
the experimental design to vary controllable and uncontrollable factors and examine how 
they affect the ability to detect and classify a diesel-electric submarine attempting to 
attack an HVU. 
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D. LITERATURE REVIEW 


A literature review is conducted to examine previous studies and documents about 
USV employment in naval operations. These studies and documents do not cover the 
scope of this thesis, but the methodologies and insights utilized in these studies are 
important to review before moving on to the model development phase. 

In her master’s thesis, Steele (2004) studies the perfonnance of a USV with 
respect to its current capabilities in information, surveillance, and reconnaissance (ISR) 
and force protection (FP) missions [11]. She uses an agent-based simulation platform 
called PYTHAGORAS to build her mission scenarios. Steele’s study explores alternative 
configurations of a prototype USV and its operational use. The results of the study 
provide some useful operational and tactical insights—ultimately, she recommends that 
the U.S. Navy use USVs in maritime missions. 

In his thesis, Abbott (2008) examines the effective use of an employed LCS 
squadron to provide analytic support for the LCS program office [12]. He builds three 
different scenarios in MANA based on the current mission packages for LCS: Anti- 
Surface Warfare (ASuW), Anti-Submarine Warfare (ASW), and Mine Warfare (MIW). 
This study touches on USVs in one of these scenarios. In the ASW scenario, a USV is 
employed to act similarly to an ASW helicopter. It is assumed that the USV has a dipping 
sonar capable of finding a submerged submarine. In this model, once a USV detects a 
submarine, it helps to localize the submarine and passes this information to an LCS for 
prosecution. With respect to the ASW scenario, the results show that sensor systems play 
a significant role. 

In 2013 the Research And Development (RAND) Corporation published U.S. 
Navy Employment Options for Unmanned Surface Vehicles (USVs) with the sponsorship 
of the Office of the Chief of Naval Operations, Assessment Division (OPNAV N81) [13]. 
This report researches the prospective suitability of USVs for U.S. Navy missions and 
functions. Firstly, it introduces the current and emerging USV marketplaces to 
understand the capabilities of platforms for U.S. Navy demands. Secondly, it develops 
concepts of employment to find out how USVs could be used in naval missions and 
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functions. It then analyzes these concepts of employment to specify highly suitable 
missions and functions. The report identities 62 potential missions and functions for USV 
employment and conducts a suitability analysis for these missions and functions based on 
pre-delined criteria. The results of this analysis show that among the 62 missions and 
functions, 27 of them are considered as highly suitable missions and functions for USV 
employment. Mostly, ASW missions fall in the category of less suitable missions and 
functions, but unarmed ASW area sanitization—a mission to detect and classify 
adversary submarines—is deemed a highly suitable mission in the emerging USV market. 
Unarmed ASW area sanitization focuses on ensuring that no enemy submarine is 
operating on transit routes or providing early warning when an enemy submarine is 
detected and classified. In this mission, USVs are deployed to an operating area ahead of 
an HVU with sufficient time to search for enemy submarines before the HVU arrives. 
USVs may conduct this mission overtly or covertly. While overt ASW operations dictate 
the use of active sonar, covert operations would use passive sonar for better concealment. 
Employing multi-mission manned platforms for this mission is expensive, both 
monetarily and in tenns of valuable resources. Reducing the risk to manned platforms 
and freeing them for other missions are the main advantages of using USVs for this 
mission. 

E. THESIS OUTLINE 

Chapter II summarizes basic concepts of ASW, informs the reader about USVs 
currently employed by the U.S. Navy, and discusses the agent-based modeling and 
simulation modeling software MANA. Chapter III explains model development and 
describes each scenario used in this thesis. Modeling assumptions and limitations are 
covered as well as agent descriptions. Chapter IV discusses the exploration of the model. 
At the beginning of this chapter, we describe the design of experiment (DOE) techniques 
that are used to investigate the simulation. Then, we explain all the controllable and 
uncontrollable factors that could potentially affect the outcome. After the discussion of 
the model exploration, the model output is analyzed using several statistical techniques, 
such as least squares regression and partition trees. Following this, factor significance is 

examined. Chapter V concludes with a summary of the thesis and provides some 
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recommendations and useful insights for decision-makers. It also includes some ideas and 
recommendations for further research. 
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II. BACKGROUND 


“The maritime should be considered as Turkey’s major national ideal and 
we have to achieve it in less time. ” 

- Mustafa Kemal Ataturk 


This chapter provides a basic operational and theoretical background on USVs 
and ASW to help guide the development of the models and discussions in this thesis. 
Since this study analyzes an ASW scenario, it is important to have some basic 
information about the concepts and components of ASW. We then provide an overview 
of technological developments of anti-submarine warfare unmanned surface vehicles 
(ASW USVs) and introduce the major missions of USVs in littoral ASW operations. We 
also provide some background on agent-based modeling and MAN A software. 

A. ANTI-SUBMARINE WARFARE 

The main purpose of ASW is to prevent our enemies from using their submarines 
effectively [14]. ASW is a branch of underwater warfare that employs a mix of naval 
platforms such as surface warships, helicopters, maritime patrol aircraft, and submarines 
to detect, track, damage, or destroy enemy submarines. In the near future, we will have 
the capability of operating a variety of unmanned vehicles in ASW operations. These 
various ASW platforms have different system and sensor capabilities. 

In order to understand the proposed model, it is important to understand the nature 
of ASW. We briefly describe littoral ASW concepts, processes, platforms, and the 
acoustic environment. 

1. Littoral ASW Concept 

In littoral waters the diesel-electric submarine remains one of the most effective 
ways to threaten operational capability. Curt Lundgren addresses the submarine threat in 
his article “Stealth in the Shallows: Sweden’s Littoral Submariners” published in Jane’s 
Navy International : 
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In the Royal Swedish Navy’s experience, the conditions make it very 
difficult to detect and prosecute a submarine. Put simply, the Baltic is an 
ASW officer’s nightmare and a submariner’s heaven. ... For an aggressor, 
submarines operating in the littoral environment are very bad news, and 
the resources and time required to find and prosecute a submarine threat 
are likely to be disproportionately high. ... The well-designed and 
proficiently crewed submarine remains a highly stealthy platfonn in the 
littoral environment. [15] 

Adversaries may conduct underwater operations on transit routes to threaten 
merchant convoys and/or HVUs. With the purpose of enabling joint or naval forces to 
conduct more successful operations, littoral ASW has to focus on denying submarine 
threats access to our areas of interest and preparing more secure spaces for friendly 
forces. In regional maritime conflicts, it is important to establish a clear battlespace and 
transit HVUs through the littoral waters [16]. 

In the near future the environment in the littoral waters will be more complex and 
chaotic due to higher density traffic and a more cluttered environment. Denying and 
eliminating stealthy submarines will be more difficult [17]. Because the littoral 
environment is very complex and noisy, traditional ASW tactics and systems optimized 
for the open ocean do not work effectively in littoral waters. High noise and poor sound 
propagation in the littoral waters negatively affect the effectiveness of the underwater 
acoustic sensors that are developed for open-ocean ASW [16]. While considering the 
special conditions in the littoral waters, there are requirements for complementary 
capabilities. A new technology insertion is a desirable approach to achieve and improve 
current and near-term ASW capabilities [16]. 

While the aim of littoral ASW operations is to detect, classify, localize, and 
neutralize adversary submarines, there will be a need to employ more capable ASW 
platforms, proficient operators, and reliable sensor systems [14]. Modern navies employ a 
variety of platforms, such as surface ships, maritime patrol aircraft, and helicopters for 
littoral ASW operations and coordinate these efforts at sea to complement ASW 
capabilities [16]. 
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2 . 


ASW Process 


Since the purpose of ASW is to eliminate the submarine threat, the ASW process 
consists of several phases. In general, this process can be simplified into five consecutive 
phases: detection, classification, localization, tracking, and kill [18]. In a typical scenario, 
ASW assets are used to detect and classify a submarine target, hold the contact, and carry 
out an accurate attack (i.e., throw weapons or depth charges), and, if necessary, regain 
contact and re-attack [19]. In this research, the effectiveness of an ASW screen formation 
is measured by the proportion of successful classifications. Therefore, we touch only on 
the detection and classification phases. These initial phases must be successful before one 
can localize, track, and attack a submarine. 

Although successful ASW requires all of these phases, the crucial and challenging 
phases are the initial detection and then classification of a submerged submarine hiding in 
the water. Once a submarine is classified, the HVU may move to avoid its weapon range. 
So, the success of an ASW operation is not only measured by the destruction of the 
enemy’s submarines [20]. Indeed, protecting the HVU is the primary ASW objective. 

a. Detection 

Detection means the observation of an underwater contact, which may be a 
submarine [18]. There are several sensors designed to detect a submarine. We divide 
these sensors into two basic categories: acoustic sensors and non-acoustic sensors. While 
acoustic sensors pick up underwater acoustic signals and transfer them into sound, non¬ 
acoustic sensors use various techniques. Acoustic and non-acoustic sensors include active 
and passive sonars, radar, magnetic anomaly detection (MAD), electronic support 
measure (ESM) devices, and sonobuoys deployed from maritime patrol aircraft (MPA). 
Visual sighting can also be a way of detecting a submarine. 

b. Classification 

For any sonar contact, the first requirement is to come to a judgement about the 
contact. This judgment is called classification [18]. Classification can be a complicated 
phase of the ASW process, but it is very important to categorize whether a contact is 
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related to a submarine or not. Contacts are classified as submarine, non-submarine, or 
doubtful. Non-submarine contacts include underwater objects such as sunken ships, sea 
creatures, downed aircraft, or lost cargo. If these underwater objects are incorrectly 
classified as submarines, it causes a waste of time and effort [18]. If a submarine is 
wrongly classified as non-submarine, the misclassification could threaten and damage 
HVUs or ASW forces. 

In tactical situations, a diesel-electric submarine operates underwater. So, it is 
important to be able to detect it there. Since overall sonar performance is degraded in the 
littorals, this platfonn gains extra stealth [21]. In practice, passive sonar is not effective in 
noisy littoral environments against diesel-electric submarines. Active sonar is the best 
available means to detect and classify this silent threat before it can launch a torpedo. 

3. ASW Platforms 

This section discusses types of ASW platforms as well as the combination of their 
properties and employment methods. A variety of platforms, including surface warships, 
rotary-wing and fixed-wing aircraft, submarines, and unmanned vehicles are used to 
localize and eliminate enemy submarines. 

There are some common capabilities that affect the success of ASW. The range or 
reach of units is an important factor in ASW as well as in ASuW and AAW. Other 
important factors are the speed and endurance of the units [14]. When an ASW 
commander makes his operations plans, he considers these factors and knows exactly the 
strengths and the weaknesses of the various ASW platforms. 

Depending on the given task, specific platforms will be assigned to form the ASW 
task force. Most often, two or more escort ships and their organic helicopters are 
expected to accompany the HVU if threatened by a diesel-electric submarine, 

a. Surface Ships 

Surface warships have many warfare capabilities other than ASW. The most 
important function of surface warships is their command, control, and communication 
capabilities. Because the payload is proportional to the size of the platform, surface 
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warships can carry a large number and variety of sensors and weapons—including other 
ASW platforms such as helicopters and USVs. 

In the littoral environment, most surface warships use their hull-mounted active 
sonars. Although surface warships such as destroyers, frigates, and littoral combat ships 
(LCSs) have a speed advantage against diesel-electric submarines, they cannot use this 
advantage effectively in ASW as speed degrades overall sonar performance [14]. 

b. ASW Helicopters 

Helicopters are widely used in ASW operations to detect and eliminate diesel- 
electric submarines hiding under temperature inversions in the water. Helicopters can be 
deployed from surface warships and extend the ships’ ASW capability. An ASW 
helicopter can operate without detection because its movements cannot be seen by the 
submarine. It can hover above the surface, lower its dipping sonar (variable depth sonar), 
and operate the sonar at a wide variety of depths (see Figure 4). In this manner they cover 
a considerable area in a short time, providing ASW helicopters a great advantage. This 
advantage is generally considered as a characteristic unique to helicopters in ASW. These 
factors provide a significant capability for ASW helicopters as a screening unit ahead of a 
HVU or naval convoy [18]. 



Figure 4. Aerial view of an SH-60F Seahawk helicopter lowering a dipping 
sonar into the Pacific Ocean (image from Wikimedia Commons 
http ://commons .wikimedia, org). 
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4. 


The Acoustic Environment 


The underwater environment is different from the surface environment. Sound 
travels unevenly through water because water is not homogenous. Both passive and 
active sonar perfonnance are significantly affected by the underwater environment. The 
measurements of temperature, pressure, and salinity all change at different layers of the 
water. The velocity and direction of sound depends on all of these factors. The changing 
acoustic conditions as a function of depth create a considerable bending effect on sound 
waves [18]. 

Temperature is the most significant variable that affects the propagation of sound 
through water. Typically, there are three layers at sea based on temperature: mixed layer 
(surface layer), thermocline, and deep water. The mixed layer is the first layer, where the 
temperature is almost constant with depth. The second layer is the thermocline, where 
temperature changes more rapidly with depth. The last one is the deep layer, where 
temperature decreases very slowly with depth [18]. The thermocline layer is the one that 
we are interested in. When sound travels into the thermocline, it tends to bend and creates 
shadow zones above and below the angle of the sound (see Figure 5). In practice, 
submarines know where the thermocline is located and use this knowledge to hide from 
surface ships. Submarines pass across the thermocline layer into and out of the mixed 
layer periodically to listen for targets. This factor gives diesel-electric submarines a great 
concealment capability. 
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Figure 5. Thermocline layer effect (image from http://weather.kopn.org). 


B. UNMANNED SURFACE VEHICLES 

Unmanned vehicles have inspired great interest and contributed considerably to 
military operations over the past two decades. This trend is likely to continue into the 
near future. Employing unmanned systems in military operations will enhance warfare 
capabilities [22]. In recent years unmanned aerial vehicles (UAVs) and unmanned 
underwater vehicles (UUVs) have benefited from significant research and development 
efforts. USVs have received relatively less focus than the other types of unmanned 
vehicles. 

1. Overview 

According to the U.S. Navy’s littoral anti-submarine warfare concept, “the 

accelerating rate of technological innovation gives increasing advantages to the navies 

that most quickly introduce appropriate new technologies into their fleets” [16]. 

According to a report of the Naval War College Global War Game in 2001, “USVs were 

key contributors in establishing situational awareness in the littorals and have shown the 

potential to provide critical access to high risk areas” [23]. In the case of possible 
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conflicts against stealthier enemies, especially in littoral waters, putting manned 
platforms at risk is no longer a reasonable course of action. US Vs are expected to be a 
critical complementary element of modern navies in the future. 

US Vs have some significant characteristics that can complement and enhance 
current warfare capabilities: reliability, maneuverability, long endurance, and high 
payload capacity. These primary features nominate the USV as a complementary element 
in multiple missions [24], Today, modern navies are looking for ways to use these risk- 
reducing platforms in naval missions, especially in littoral waters. 

In 2007, the U.S. Navy published “The Navy Unmanned Surface Vehicle (USV) 
Master Plan” [1]. This master plan examines the capabilities, classes, and potential naval 
missions for USVs. Seven high-priority USV missions are identified in the master plan. 
These missions, in priority order, are [1] 

• Mine Countermeasures 

• Anti-Submarine Warfare 

• Maritime Security 

• Surface Warfare 

• Special Operations Forces Support 

• Electronic Warfare 

• Maritime Interdiction Operations Support 

According to open online sources, the U.S. Navy currently has four classes of 
USVs. These are Fleet Class I, Semi-Submersible Snorkeling Vessel, Harbor Class, and 
Small Class [25]. Their primary missions are antisubmarine, mine countermeasures, and 
surface warfare missions for the littorals. 

2. Development of the Anti-Submarine Warfare Unmanned Surface 

Vehicle 

In recent years, advances in defense technologies have offered a variety of 
payloads and systems for USV applications. Potential payloads for USV systems include 
towed array sonars, dipping sonars, and acoustic sensors. A compact dipping sonar 
system is now optimized for the USV. Therefore, a USV can take advantage of the same 
sensor capability as ASW helicopters. 
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General Dynamics Robotic Systems (GDRS) developed an 11 meter “Fleet” class 
Anti-Submarine Warfare Unmanned Surface Vehicle (ASW USV) for use on the LCS 
and delivered the first one to the U.S. Navy in 2008. The ASW USV is autonomous and 
capable of operating in an extended-duration with a high-payload capacity. It has high 
speed capability (35+knots), thus it can expand the reach of surface warships. 
Characteristics of this ASW USV are shown in Table 1. 

Table 1. Principal characteristics of anti-submarine warfare unmanned 

surface vehicle (ASW USV). 


Characteristic 


Characteristic 


Length 

40 ft 

Payload 

5000 lb 

Beam 

11.2ft 

Max Speed 

35+ kt 

Max Weight 

21,120 lb 

Endurance 

24+hr 


3. USV Employment for Antisubmarine Warfare 

Today’s ASW techniques are effective in most cases, but employing USVs is 
likely to increase the effectiveness of ASW. Employing USVs in littoral ASW operations 
has potential to enhance the efforts of existing ASW assets. Effective employment and 
the correct tactical use of USVs offers a great force multiplier. 

U.S. Navy USV Master Plan (2007) defines littoral ASW missions in three major 
categories (see Figure 6): “Hold at Risk,” “Maritime Shield,” and “Protected Passage” 
[!]• 
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Figure 6. Littoral ASW missions in three major categories. 


• In a Hold at Risk scenario, USVs monitor for submarines in the entrance 
of ports or chokepoints, but they are not the ideal candidate for this 
category due to their limited stealth. 

• Maritime Shield missions focus on clearing a Carrier Strike Group (CSG) 
or Amphibious Ready Group (ARG) operating area from adversary 
submarines and keeping that area secure. 

• In a Protected Passage scenario, USVs clear the battlespace of enemy 
submarines to enable secure routes for an Expeditionary Strike Group 
(ESG) or HVU. 

In all the scenarios, USVs reduce the risk to manned platforms and serve as 
offboard sensors, thereby extending the reach of warships. A warship can launch a USV 
and serve as its mother ship. 

C. AGENT-BASED MODELING 

Agent-based modeling is a simulation modeling technique that has been used 
extensively in solving real-world problems, including military applications [26]. In agent- 
based modeling, we simulate multiple autonomous decision-making entities called 
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agents. Each agent makes its own decisions on the basis of a set of user defined rules and 
behaves stochastically in its local environment [27]. Agents can determine their behaviors 
with their predefined personalities and be aware of events or other agents by using 
organic or inorganic sensors. 

Agent-based models can perfonn non-linear behavior patterns, capture 
organizational dynamics, and provide valuable insights about real-world systems [26]. 
Military applications of agent-based simulations are widely used in the decision-making 
process. Agent-based simulations can capture the more chaotic and intangible aspects of 
military conflicts. These simulations assist decision-makers in testing war plans, 
reviewing or proposing force structures, providing detailed information on today’s high 
technology products, deciding how to use sensors and weapons, and exploring potential 
changes in doctrine or tactics [28], 

There are many simulation tools that are widely used for agent-based modeling. 
These tools include general computational mathematics systems such as MATLAB and 
Mathematica; general programming languages such as Python, Java, C++, and C; and 
other agent-based modeling platfonns such as NetLogo, Swarm, Repast, AnyLogic, 
JANUS, MAN A, and Pythagoras [29]. These tools are used in different fields of study 
and real-world applications. Figure 7 displays a screen shot of a USV scenario in 
Pythagoras [11]. 
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Figure 7. A screen shot of a USV scenario in Pythagoras, from [11]. 


D. MAP AWARE NON-UNIFORM AUTOMATA (MANA) 

The simulation tool used in this thesis is MANA, which is developed by the 
Defence Technology Agency in New Zealand. MANA has been widely used for military 
and academic studies, including several master’s theses at the Naval Postgraduate School. 
These studies include maritime protection of critical infrastructure assets [30], counter¬ 
piracy escort operations in the Gulf of Aden [31], unmanned aerial vehicle contributions 
for expeditionary operations [32], the effectiveness of unmanned aerial vehicles in 
helping secure a border [33], and the operational effectiveness of a small surface combat 
ship in an anti-surface warfare environment [34]. 

MANA is designed for modeling complex adaptive systems, such as combat 
situations. MANA builds time-stepped, mission-level, stochastic simulations. MANA 
contains entities representing military units which interact with their environment and the 
other entities and make their own decisions. Unlike physics-based models, MANA is 
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very useful to simulate and analyze the effects of command and control, situational 
awareness, and sensor and weapon systems [35]. Figure 8 shows the startup screen of 
MANA. 



MANA V 


DTA 


.★ 
★ ★ 


DEFENCE TECHNOLOGY AGENCY 


Vector-Based Movement 

Map Aware Non-uniform Automata 


This software has been developed by Gregory McIntosh . David Galliqan . 

Mark Anderson and Michael Lauren for the New Zealand Army and Defence Force. 

Terms of use: 

This software belongs to the Defence Technology Agency (DTA), New Zealand 
Defence Force. It is not to be distributed without the permission of DTA. This 
product is not to be offered for resale. 


View end user license agreement (EULA1 




See https:;/teams.nzdf.mil.nz/sites/mana for updates and support. 

(Requires user account.) 


I understand I can only use this software subject to the conditions stated above. 
Version 5.01.09 


I Agree 


Cancel 


pic: J Peefc 


Figure 8. The startup screen for MANA. 


MANA modelers have the ability to edit battlefield characteristics and create a 
terrain map and background according to specific scenarios. Agents behave 
independently on the virtual battlefield based on their personalities, goals, sensors, 
weapons, and terrain type. However, they will not respond to the situations in the same 
way because the platfonn is stochastic and each agent uses its own information provided 
by personal sensors or communication li nk s and stored in organic/inorganic SA maps. 
Agents can also have completely different personalities in different states and behave in 
that way by activating trigger states. 
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MANA Version 4 User Manual defines four basic parameters that affect an agent’s 
behavior [36]: 

• Personality weightings determine an agent’s tendency to move towards or 
away from friendly, neutral, or enemy entities, or waypoints, or terrain. 

• Move constraints are meta-personalities which modify an agent’s basic 
personality weightings. This brings an agent a detailed behavior ability 
which is closer to the reality. 

• Intrinsic capabilities are tangible or physical characteristics of an agent 
including its speed, sensors, weapons, targeting priorities, and fuel level. 

• Movement algorithm modifiers affect an agent’s speed and degree of 
autonomy when moving. 

More infonnation can be found in MANA Version 4 User Manual and MANA-V 
(Map Aware Non-uniform Automata-Vector) Supplementary Manual. 
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III. MODEL DEVELOPMENT 


“We are entering an era in which unmanned vehicles of all kinds will take on greater 
importance in space, on land, in the air and at sea. ” 

- George W. Bush 

In this chapter, a brief description of ASW screen formation is given, as well as 
the scenarios used for this thesis. After addressing the scenarios, we discuss some key 
modeling assumptions and limitations. Finally, measures of effectiveness and model stop 
conditions are explained. 

A. ANTI-SUBMARINE WARFARE SCREEN FORMATION 

The purpose of defensive ASW operations is to protect a convoy of ships or 
HVUs within a group through high-threat areas. Because conventional submarines are 
serious threats to HVUs in the littoral waters, naval operations usually focus on defensive 
ASW. HVU protection requires detailed organization and a carefully set formation. A 
defensive ASW formation (see Figure 9) is generally used for preventing a submarine 
from reaching a position around an HVU from which it could launch a torpedo. It is 
necessary to use acoustic equipment effectively by employing highly maneuverable 
surface craft, such as destroyers and frigates, and helicopters at an effective distance from 
an HVU or a convoy of ships. This formation is generally called an ASW screen 
formation. 

The screen size depends on the availability of screening vessels in the ASW task 
force. If a large force is available, two or three screens may be employed in the 
formation. One or two screens are normally used for small forces. There are three classes 
of ASW screens [37]: 

• The inner ASW screen is a screen in which surface ships position around 
an HVU or convoy for the purpose of preventing a submarine from 
reaching the torpedo danger zone. 
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• The intermediate ASW screen is a second screen that is farther away from 
a formation of ships, has the potential to enhance detection and 
neutralization capabilities. 

• The outer ASW screen is a sound screen well ahead of the formation of 
ships and HVU for the purpose of detecting the approach of a submarine 
and alerting the assets early. 


Example: 

"Operation 

Atlantic” 

Planning 
TG 2(H). 1 

Screen Units: 

2 frigates 

1 Destroyer 

2 Helicopters 


Figure 9. Possible ASW screen formation, from [38]. 

The inner ASW screen is the most important one among these three classes. The 
form of the inner ASW screen is shaped based on the number of available screening 
ships. The outer screen is the next most important one, and ASW helicopters are 
generally used for it. If screening vessels exceed the number required for the inner and 
outer ASW screens, the intermediate ASW screen may be employed. 

B. SCENARIO DESCRIPTIONS 

This thesis uses the combat simulating platform called MANA to model the 
scenarios. In this section, the battlefield features are briefly explained. Then, a generic 
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ASW scenario is created to increase to facilitate exploring USV capabilities and tactics. 
Next, we describe all of the scenarios. 

1. The Battlefield 

The battlefield is configured as 40 nautical miles (nm) wide by 140 nm long. On 
this battlefield, our area of interest is a 100 x 24 nm box in which MANA positions the 
enemy submarine randomly. The entire battlefield is plain terrain; thus, the terrain has no 
effect on the movements of the agents. In this model, the Cartesian coordinate system 
describes all positions in the battlefield. For all scenarios, the top left-hand comer of the 
battlespace is point (0, 0), and the bottom right-hand corner is point (140, 40). The 
battlefield characteristics are shown in Figure 10. 

2. Generic Scenario 

A Turkish naval task force (Blue) has been tasked to move from an area of 
operation to another. The aim of this task force is to transport logistics to friendly forces 
operating at sea. This task force consists of guided-missile aviation frigates (FFGH), 
ASW helicopters (SH-70B), and unmanned surface vehicles (USVs). Their main goal is 
to protect the HVU, a mid-size replenishment oiler (AOR). Helicopters and USVs are 
organic to the frigates. These assets can be deployed from the frigates and generally 
operate ahead of the task force. 

Intelligence reports warn that an adversary (Red) diesel-electric submarine 
threatens the SLOCs. It is assumed that this enemy submarine is on Blue’s transit routes, 
waiting for a favorable moment to engage the HVU with a torpedo. The submarine 
selects its target carefully; it almost never launches a torpedo blindly into the task force. 
It is assumed that an attack on ASW assets is never expected because the diesel-electric 
submarine desires the more strategic oiler and an attack on an escort will alert its primary 
target. That is, the submarine will not put its life at risk unless it can fire at the HVU. 

An ASW screen is fonned to detect and classify a submarine when a task force is 
transiting high-threat areas. The deployment tactic plays an important role on detection 
and classification of the submarine. The ASW assets try to detect and classify the 
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submarine before it penetrates the screen, takes a planned approach, and launches a 
torpedo. Once the submarine is classified, normally, the ASW task force attempts to 
execute the localization, tracking, and kill phases. However, this scenario focuses solely 
on classifying the submarine before it enters the torpedo danger zone (TDZ) around the 
HVU. Classifying the enemy submarine can be interpreted as reducing the risk to the 
HVU. In this study, the ASW process after the classification phase is not simulated. The 
overall representation of the generic scenario is depicted in Figure 10. 



Figure 10. The battlefield characteristics and the overall representation of the 

generic scenario (not drawn to scale). 


The baseline and advanced scenarios are modeled using MANA. The scenarios 
were built to explore the use of combinations of frigates, helicopters, and USVs to protect 
an HVU from a single enemy submarine. In the scenario setup, the number of available 
frigates ranges from two to three. The number of helicopters and USVs are dependent on 
the number of frigates, which serve as mother ships to helicopters and USVs. The overall 
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scenario description is shown in Table 2. The modeling process is explained in simple 
language in the following sections. 


Table 2. The overall scenario description. 


Scenario 

ASW Units 

Baseline Scenario 

2FFGH 

2HELO 

- 

Scenario Two 

2FFGH 

- 

2 USV 

Scenario Three 

2FFGH 

2HELO 

2 USV 

Scenario Four 

3 FFGH 

3 HELO 

- 

Scenario Five 

3 FFGH 

- 

3 USV 

Scenario Six 

3 FFGH 

3 HELO 

3 USV 


3. Baseline Scenario 

This scenario is created based on existing ASW screening settings. It provides us 
a standardized benchmark. There are four classes of agents in the battlespace: the HVU, 
frigates, ASW helicopters, and the enemy submarine. In the baseline scenario, the HVU 
is screened by two frigates and two organic ASW helicopters because it does not have an 
ASW capability, and it is vulnerable to submarine attacks. The frigates are equipped with 
hull-mounted sonars, and the ASW helicopters are equipped with dipping sonars. All 
equipped vessels are using their sonars in active mode. The submarine listens for sound 
in passive mode. 

While the frigates are positioned on the inner ASW screen, the helicopters are 
positioned on the outer ASW screen. The initial locations of the units are defined using 
Cartesian coordinates. The ASW assets are initially located at the western edge of the 
battlefield outside the box. The coordinates of the battlefield, the area of interest, and the 
initial locations of the units are depicted in Figure 11. 
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Figure 11. The coordinates of the battlefield, the area of interest, and the initial 
locations of the units for the baseline scenario (not drawn to scale). 


The HVU begins at the point (2, 20) and proceeds as a moving reference point at 
10 knots, which is the speed of advance (SOA). The frigates maintain this speed, and 
their movements depend on the HVU. The helicopters are initially stationed on the 
frigates. They launch from their mother ships and move to the first dip location once the 
simulation starts. Once there, they hover in place and lower their sonar transducers into 
the water. 

MANA randomizes the initial positions of the agents within their defined 
homeboxes. Therefore, we can expect different outcomes each time the model is run. At 
initialization, the diesel-electric submarine is positioned randomly by MANA in the area 
of interest and thereafter moves randomly at 3 knots. When it becomes aware of the task 
force, it attempts to penetrate the ASW screen and increases its speed up to 10 knots. 
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4. 


Scenario Two 


In this scenario, USVs are included in an intermediate screen instead of the outer 
screen ASW helicopters. In doing so, USVs will maintain a protective ASW barrier in 
front of the surface group. Referencing the coordinate system in Figure 11, the starting 
locations of the units are shown as follows: 

• BlueHVU: (2,20) 

• BlueEscortl: (7,26) 

• BlueEscort2: (7, 14) 

• BlueUSVl: (19,25) 

• BlueUSV2: (19, 15) 

USVs carry a dipping sonar similar to the one used by the helicopters. The USVs 
use a “Sprint & Drift” tactic ahead of the mother ship. They sprint ahead to their next dip 
location, and once there, they drift on the water and lower and operate their dipping 
sonar. 


5. Scenario Three 

In Scenario Three, we update the baseline scenario again. In this scenario, all of 
the available assets are deployed: two frigates, two ASW helicopters, and two USVs. All 
of the agents are using the same tactics previously discussed. While ASW helicopters are 
positioned on the outer ASW screen, USVs are positioned on the intermediate ASW 
screen. The HVU, frigates, and USVs are located at the same starting locations as in 
Scenario Two. Once the simulation starts, the helicopters are deployed ahead of the 
USVs. 


6. Scenario Four 

In this scenario, the HVU is screened by three frigates and three organic ASW 
helicopters. There is no difference between this scenario and the baseline scenario in 
tenns of the deployment tactics and parameter setup, but the number and placement of 
the units change. The initial locations of the units are shown in Figure 12. 
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Figure 12. Scenario Four: The initial locations of the units (not drawn to scale). 


7. Scenario Five 

In Scenario Five, USVs are deployed again in our model instead of ASW 
helicopters. The deployment tactics and parameter setup are the same as before, but the 
initial locations and the sectors relative to the HVU are different. Referencing the 
coordinate system in Figure 12, the starting locations of the units are shown as follows: 

• BlueHVU: (2,20) 

• BlueEscortl: (7,28) 

• BlueEscort2: (7, 12) 

• BlueEscort3: (8, 20) 

• BlueUSVl: (18,28) 

• BlueUSV2: (18, 12) 

• BlueUSV3: (18,20) 

8. Scenario Six 

In Scenario Six, all of the available assets are deployed: three frigates, three ASW 
helicopters, and three USVs. All of these agents act in the same manner as in previous 
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scenarios. The HVU, frigates, and USVs are located at the same starting locations as in 
Scenario Five. Once the simulation starts, the helicopters are deployed on the outer ASW 
screen ahead of the USVs in an intennediate screen. 

C. AGENT DESCRIPTIONS 

MANA agents have a variety of tangible characteristics, such as agent allegiance 
(friendly, enemy, or neutral), class parameters, threat levels, movement speed, and 
personal concealment rate. 

The basic assessment of an agent’s identity is that of allegiance. Allegiance 
determines the side of an agent. We define the allegiance of the HVU, frigates, 
helicopters, and USVs as friendly, and the allegiance of the diesel-electric submarine as 
enemy. There are no neutral agents in our scenarios. We also added stationary dummy 
agents that simulate random dipping locations for helicopters and USVs. Their allegiance 
is defined as enemy for modeling purposes as they “attract” the helicopter and USV 
agents. The numeric value 1 represents blue forces and 2 represents red forces. 

Agent class parameters and agent threat levels help define the type of the enemy. 
Agent class is used to differentiate the target types for weapon engagement. Because we 
do not simulate the kill phase in this model, a dummy weapon model is used for stopping 
the simulation when the submarine is classified. 

The threat level is used to differentiate the target types on the situational 
awareness maps of the agents, so the agent can react to that information according to user 
assigned personality weightings. Table 3 shows the overall tangible characteristics of the 
agents. 
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Table 3. The tangible characteristics of the agents. 


Agent 

Description 

Allegiance 

Agent Class 

Threat 

HVU 

High Value Unit 

1 

1 

3 

Escort Ship 

Guided-Missile Aviation 
Frigate (FFGH) 

1 

2 

1 

Helicopter 

ASW Helicopter 

1 

3 

1 

USV 

USV equipped with 
dipping sonar 

1 

3 

1 

Submarine 

Conventional Diesel- 
Electric Submarine 

2 

4 

3 

Dipping Agent 

Dummy Enemy Agent 

2 

94-99 

2 


1. Friendly Forces B ehaviors 

The movement behavior of an ASW unit is based on its personality weightings 
and next waypoint. In MANA, the personality weightings are set between -100 and 100 
for adjusting the directivity of the agent. For more details, see the MANA Version 4 User 
Manual [36]. A positive weighting value attracts an agent while a negative value repulses 
it. The modeler can play with the weighting values to obtain the desired behavior. 

a. HVU and Escort Ships 

For defining the movement behavior of the FIVU and the escort ships, their 
personality weightings towards the next waypoint are set to 100. Their movement toward 
the waypoint is slightly randomized by setting the random patrol bar to 10. This adds a 
small amount of random wiggle to their movement, as with real platforms. The FIVU uses 
just the Default State settings because it does not change its behavior during the 
simulation. Figure 13 shows the personality weightings and trigger states of the FIVU. 
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Figure 13. The personality weightings and trigger states of the HYU. 


The escort ships have four states created to simulate helicopter operations. We did 
this by applying the embussing feature in MANA. The helicopters are carried by the 
escort ships until a release trigger point is reached. In the beginning of the simulation, the 
escort ships change their state from Default State to Run Start. In this state, they release 
their child squads, that is, their organic helicopters. After the duration time of the trigger 
state passes, the escort ships’ states fall back to the Must Embuss state. In this state, the 
escort ships call their child squads back. After the child squads arrive at the escort ships, 
they station there during the Embussecl Children state. This process is used to model the 
endurance of the helicopters. Table 4 shows the trigger states of the escort ships. 
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Table 4. The trigger states of the escort ships. 


State 

Embussing Behavior 

Next State 

Default 

Nothing 

Run Start 

Run Start 

Release Child Squads 

Must Embuss 

Must Embuss 

Embuss Children 

Embussed Children 

Embussed Children 

Nothing 

Run Start 


b. Helicopters and US Vs 

The helicopters and USVs have the same movement pattern. They first move to 
the nearest dipping location, which is semi-randomized in the area of interest. Dummy 
enemy agents are created to simulate dipping locations and randomized in their 
homeboxes. The homeboxes are set to a reasonable search pattern. The threat level of 
these agents is set to 2, and they attract the helicopters and USVs in their Default State. 
Once a helicopter or USV finds the nearest dummy agent, it fires at this agent, and then 
changes its state to Taken Shot (Sec). In this state, a helicopter hovers over the water and 
lowers its dipping sonar for four minutes. Next, its state falls back to Spare 1 during 
which it enables its dipping sonar. Then, it recovers its dipping sonar for four minutes 
and moves forward to find the next dummy agent. Table5 summarizes this process. 


Table 5. The trigger states of the helicopters and USVs. 


State 

Speed (knots) 

Enable 

Sensor? 

(Yes/No) 

Duration 

(seconds) 

Next State 

Helicopter 

USV 

Default 

Flight 

Speed 

Sprint 

Speed 

No 

Sprint Time 

Taken Shot 
(Sec) 

Taken Shot (Sec) 

0 

0 

No 

240 

Spare 1 

Spare 1 

0 

0 

Yes 

Dip Time 

Spare 2 

Spare 2 

0 

0 

No 

240 

Default 
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2. Enemy Behaviors 

The submarine behavior is a simple process. It has two states: Default and Enemy 
Contact. In the default state, it patrols in its homebox and tries to detect ASW units. A 
patrol zone is created by using the random patrol feature in MANA. This allows the 
submarine to travel on a straight path on random routes in the patrol zone. The random 
patrol settings of the submarine are shown in Figure 14. 


Random Patrol 
10 « “□ V 

Average path length: 

5000 g metres 

[V Define Patrol Zone 

Left = 20.0000 n. miles 
Top = 5.0000 n. miles 
Width = 100.0001 n. miles 
Height = 24.0000 n. miles 

Figure 14. The random patrol settings of the submarine. 

Once the submarine detects an ASW unit, it changes its state to Enemy Contact. It 
then moves through the center of the formation and attempts to reach the TDZ. The 
submarine moves forward in a submerged approach region and attempts to remain 
undetected to reach the TDZ. This movement is set with several changes in personality 
settings of the agent. The personality settings of the submarine in Enemy Contact state 
are shown in Figure 15. 
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Figure 15. The personality settings of the submarine in enemy contact state. 
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3. Sensor Behaviors 

In the model, sonar is the only detection sensor used by the agents since the 
submarine is submerged. The escort ships use their hull-mounted sonar while the 
helicopters and USVs use their dipping sonar in active mode to detect the submarine. The 
submarine uses its hull-mounted sonar in passive mode to detect the ASW assets. While 
an advanced (probabilistic) sensor model is used to model the active sonar of ASW units, 
a cookie-cutter sensor model (see Figure 16) is used to model the passive sonar of the 
submarine. 
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Figure 16. Cookie-cutter sensor. 

Two different sensors are modeled as cookie-cutter sensors for the submarine’s 
passive sonar. One sensor type is modeled for counter-detection of ASW units that 
operate active sonar. The other sensor type is modeled for detecting the HVU, which does 
not have any acoustic sensors. Because counter-detection of active sonars can be 
performed at greater ranges than the passive sonar’s detection range of the HVU, the 
counter-detection range is fixed at 18,288 meters (20,000 yards) in this model while the 
detection range is fixed at 10,973 meters (12,000 yards). A cookie-cutter sensor detects 
all contacts within its maximum range. Once an ASW asset enters the detection range of 
the submarine, MANA records the detection and classification with the probability of 1.0. 
The sensor models are visualized in Figure 17. 
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Figure 17. Sensor models. 

In MANA, sensor models for the advanced sensor type are defined with a 
detection range-time table and the classification range-probability table [36]. The 
detection range-time table defines the average time between detections in seconds for the 
specified sensor detection range. Figure 18 shows an escort ship’s sensor setup panel. In 
this panel, the detection range of the frigate is set to 10,973 meters (12,000 yards). If an 
underwater contact moves in the detection range of the sensor, the frigate will detect this 
contact on average every 300 seconds—with a random draw each time step. For every 
detection event, the submarine has a chance to hide in the water based on its personal 
concealment rate. Once a contact has been detected, the ASW unit has to categorize 
whether the contact is related to a submarine. Detection is a required event for the 
classification process to occur. 

The classification range-probability table determines the probability of classifying 
the contact for the specified classification range once the detection event occurs. In 
Figure 18, the classification range is set to 7,315 meters (8,000 yards). This means that if 
the submarine is in this range, the escort ship has a chance to classify it. If the submarine 
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is in the detection range, but out of the classification range, the detection may be 
successful, but the classification will not be. 



Figure 18. Setup panel for an advanced sensor model. 


Detection ranges and classification range intervals for all the agents are 
summarized in Table 6. 
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Table 6. Sensor detection ranges and classification range intervals. 


Sonar Type 

Detection 

Range 

Classification Range 
Interval 

Minimum 

Maximum 

Ship hull-mounted sonar 

12000 yards 

6000 yards 

10000 yards 

Dipping sonar 

12000 yards 

4000 yards 

10000 yards 

Submarine 

hull-mounted 

sonar 

Counter-detection 

20000 yards 

20000 yards 

Detection 

15000 yards 

15000 yards 


D. STOP CONDITIONS 

Stop conditions were introduced to the model to reduce runtime. The simulation 
stops when one of the following conditions happens (see Figure 19): 

• The submarine is classified by one of the ASW units; 

• The submarine reaches the TDZ around the HVU; or 

• The FIVU reaches its final waypoint. 



Figure 19. Stop conditions. 
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E. SCENARIO ASSUMPTIONS AND LIMITATIONS 


Scenario assumptions and limitations are vital to a successful study. It is 
necessary to make acceptable assumptions and define limitations to create a model 
realistic enough to obtain useful insights. 

1. Assumptions 

a. Friendly Forces 

(1) USVs are launched from surface warships and they are fully autonomous. 

(2) USVs meet autonomous requirements, such as station-keeping. 

(3) Helicopters and USVs use a Sprint & Drift tactic. They sprint ahead to the 
next dip location, and once there, they drift or hover on the water and 
lower and operate their dipping sonar. 

(4) Dipping points are semi-randomized in the area of interest. 

(5) Once an ASW unit detects an underwater contact, it can execute the 
classification process itself. 

(6) Each unit has a chance of classifying the submarine for every detection 
event. 

b. Enemy 

(1) The submarine operates submerged during the simulation. By doing this, it 

minimizes detection by the ASW forces. 

(2) The submarine’s initial position is selected at random in its homebox. 

(3) The submarine does not attack frigates, helicopters, or USVs. Its only 

target is the HVU. 

2. Limitations 

We defined the limitations of MANA when building the model. These limitations 
must be considered in the analysis chapter. The first and most important one is that it is 
very hard to implement an advanced naval formation, such as an ASW screen fonnation, 
in MANA. Another limitation is that the level of classification is limited to a binary 
response: 0 or 1. For us, 0 represents the levels non-submarine (NONSUB) and doubtful 
(POSSUB and PROBSUB) levels, and 1 represents the certain submarine (CERTSUB) 
level. Also, it is difficult to simulate the underwater environment. The changes in the 


43 



environmental conditions are simulated by varying detection chances and classification 
probabilities. Finally, the submarine’s actual depth is not explicitly simulated. The 
submarine can hide below thennal layers, beneath undersea mountains, or on the sea 
floor. The submarine’s concealment rate per detection event accounts for the submarine’s 
stealthiness. 
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IV. MODEL EXPLORATION 


A. DESIGN OF EXPERIMENTS 

The Design of Experiments (DOE) is a practical approach for large-scale 
experiments to examine design factors and determine the relationship between design 
factors and output responses. In experimental tenninology, design factors are the input 
variables, and output responses are the measures of effectiveness or performance [39]. 

Although cluster computers can run simulations very quickly, it is an impossible 
task to run all possible design points. The quality of the results can be determined by the 
model runs. An efficient design is needed to analyze a sufficient breadth of possible 
outcomes. Otherwise, we may limit the insights in the analysis. 

In this thesis, a nearly orthogonal Latin hypercube (NOLH) spreadsheet 
developed by Susan Sanchez is used to generate the design points [40]. The advantage of 
using an NOLH design is that it has good space-filling properties and meets the 
orthogonality criteria necessary for good statistical properties of analysis methods. This 
design can provide efficient information about the experiment. A well-designed NOLH 
allows the analyst to efficiently explore more factors across the design space and fit a 
variety of diverse models to multiple different response variables. A scatterplot matrix in 
Ligure 20 shows the space-filling properties of our NOLH design. 

The NOLH design spreadsheet allows us to create an efficient design and saves 
time and effort. Different designs are available on this spreadsheet based on the number 
of design factors. A design with more design points is a favorable thing, but not required. 
We choose the 16-factor design to build our experimental design. The design points used 
in this study are provided in Appendix A. 
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Figure 20. Scatterplot matrix for the design factors. 


B. DESIGN FACTORS 

In an ASW scenario, many factors may affect the outcome. In total, 16 factors 
were varied in the simulation for each scenario. Design factors can be divided into two 
groups: controllable and uncontrollable factors. These factors are varied over a range in 
order to explore their effects on the outcome. They are explained in the following 
sections. Table 7 shows the list of factors with their ranges, units, and explanations. 
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Table 7. Description of controllable and uncontrollable factors. 


Factor 

Explanation 

Minimum 

Maximum 

Unit 

Controllable Factors 

FleloSpeed 

Flelicopter Speed at Default 

State 

80 

120 

knot 

USVSpeed 

USV Speed at Default State 

20 

35 

knot 

FleloAvgDetTime 

Flelicopter Sonar Average Time 
Between Detections at Default 
State 

180 

360 

second 

USVAvgDetTime 

USV Sonar Average Time 
Between Detections at Default 
State 

180 

360 

second 

FleloEndTime 

Flelicopter Endurance (Duration 
Time of Escort Ships’ Run Start 
State) 

5400 

9000 

second 

FleloRefTime 

Flelicopter Refuel Time 
(Duration Time of Escort 

Ships’ Embussed Children 

State) 

1200 

2400 

second 

FleloDipTime 

Flelicopter Dipping Time 
(Duration Time of Flelicopters’ 
Spare 1 State) 

300 

600 

second 

USVDipTime 

USV Dipping Time (Duration 
Time of USVs’ Sparel State) 

300 

600 

second 

FrigateClassify Range 

Frigate Sonar Classification 
Range 

5486 

9144 

meter 

FleloClassify Range 

Flelicopter Sonar Classification 
Range 

3658 

9144 

meter 

US VC lassify Range 

USV Sonar Classification 

Range 

3658 

9144 

meter 

FrigateClassify Prob 

Frigate Classification 

Probability 

0.2 

0.4 

- 

FleloClassify Prob 

Flelicopter Classification 
Probability 

0.4 

0.6 

- 

U S V Classify Prob 

USV Sonar Classification 
Probability 

0.4 

0.6 

- 

Uncontrollable Factors 

SubAttackSpeed 

Submarine Attack Speed at 
Enemy Contact State 

5 

10 

knot 

SubConcealment 

Submarine Personal 

Concealment per Detection 

40 

60 

% 
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Controllable Factors 


Controllable factors are related to the decisions of friendly assets, which can be 
decided upon in advance or during the mission. In this model, they are all related to the 
characteristics of the frigate, helicopter, and USV. Controllable factors included the 
movement speed, dipping time, endurance time, refuel time, average time between 
detections, classification range, and classification probability. 

a. Movement Speed 

The movement speed is varied in the experimental design to determine whether 
this factor has an effect on mission success. Because the most considerable strength of 
the helicopter is its high speed capability, it can execute the dipping process easily over a 
large area. However, its endurance is limited. Once the helicopter runs out of fuel, it 
moves back to the mother ship, refuels, and deploys to the station again. Four states are 
defined to simulate the dipping process. In “Default State,” a helicopter moves to the next 
dipping location with a speed of 80 to 120 knots. In a trigger state, the speed of the 
helicopter is set to zero since it hovers over the water. 

The helicopters are significantly faster than the USVs. This factor may give the 
helicopters an advantage over USVs in terms of mobility. USVs execute the dipping 
process the same way as the helicopter. When a USV is in “Default State,” it moves to 
the next dipping location with a speed of 20 to 35 knots. In a trigger state, it drifts on the 
water with zero speed. 

b. Sensors 

The main focus of the defensive ASW operations is the ability to detect and 
classify the enemy submarine. Since using active sonar is the primary method to detect 
and classify the diesel-electric submarine, the frigate uses a hull-mounted sonar, and the 
helicopter and USV use dipping sonars in active mode. The sensor parameters are mostly 
controllable because the selection of the sensor type and capability can be decided during 
the design process. But, the effectiveness of a sonar is limited by environmental 
conditions. In our model, we consider the sensor parameters only partly controllable. 
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The helicopter and USV use the same sensor type in the model. The perfonnance 
of the sonar for each platform can be evaluated by varying three factors: the time interval 
between consecutive detections, classification range, and classification probability. These 
factors relate to the helicopter and USV varied independently over the same range. So, 
we can explore three different cases: a better sonar perfonnance for the USV, a better 
performance for the helicopter, and the same perfonnance for both platforms. 

We define the time interval between consecutive detections as the period between 
the event initiation and completion. This is a simplification of simulating detection 
chances. The other states are locked to the default state, so varying the average time 
between detections in the default state is enough. Thus, it is desirable to have the mean 
detection time as small as possible. 

The classification ranges of the platforms provide a reduced danger area. If the 
platforms have a short classification range, then the submarine has a good chance of 
penetrating the ASW screen. In ASW, the classification range depends on underwater 
conditions, background noise in the ocean, and sonar capability. 

The other sensor parameter is the classification probability. In ASW, this factor 
may depend on target characteristics or the training of the operators. For USVs, it may 
depend on the development and perfonnance of automatic detection and classification 
systems and techniques. 

c. Tactical Employment of ASW Assets 

Six scenarios were built to explore the use of the combinations of frigates, 
helicopters, and USVs. The name of the scenarios is viewed as a categorical factor for a 
quick comparison of the scenarios. We expanded this factor into three different 
categorical factors for partition tree and regression analysis: the number of available 
frigates, helicopter presence, and USV presence. These factors relate to the tactical 
employment of the helicopters and USVs as well as the design of the ships. The factors 
related to the scenarios are shown in Table 8. 
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Table 8. The factors related to scenario setup. 


Scenario 

The Number 
of Available 
Frigates 

Helicopter 
Presence 
(l=Yes, 0=No) 

USY Presence 
(l=Yes, 0=No) 

Baseline Scenario 

2 

1 

0 

Scenario Two 

2 

0 

1 

Scenario Three 

2 

1 

1 

Scenario Four 

3 

1 

0 

Scenario Five 

3 

0 

1 

Scenario Six 

3 

1 

1 


2. Uncontrollable Factors 

Uncontrollable factors are related to the enemy and uncertainty in the combat 
environment. There are two factors regarding the enemy submarine: attack speed and 
personal concealment per detection opportunity. 

a. Speed 

The submarine patrols at 3 knots in its homebox. When it detects an ASW unit, it 
changes its state to “Enemy Contact State” and tries to penetrate the ASW screen by 
increasing its speed. The speed of the submarine in “Enemy Contact State” is a factor that 
ranges from 5 to 10 knots. 

b. Stealth 

The submarine is designed to submerge and maneuver quietly to avoid detection. 
In a noisy littoral environment, the submarine gains extra stealth. The submarine can find 
shadow zones to hide from active sonar and approach an enemy without being detected. 
This factor was simulated in MANA using the personal concealment per detection 
feature. This factor represents a probability of stealth per detection event. The stealth of 
the submarine is varied between 40% and 60% in the experimental design. 
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C. DATA ANALYSIS 

After explaining the model development, experimental design, and design factors, 
we now focus on data analysis. In this section, model runs, our analysis tools, and 
measures of effectiveness are described briefly. This is followed by a comparison of the 
scenarios, regression analysis, and partition trees. 

1. Model Runs 

Using the NOLH design for 16-factors, 65 design points were created for each 
scenario. Each design point was run 1,000 times, resulting in 65,000 simulated ASW 
missions for each scenario. The time step in this model is fixed to one second because it 
was observed that large time steps led to unusual behaviors. Since there are six different 
scenarios being evaluated, a total of 390,000 runs were executed. On average, each model 
run takes approximately one minute of computer runtime on a personal computer. As 
expected, the more agents that are included in the model, the longer the runtime it takes. 
For example, the scenarios with three frigates take more time to run than the scenarios 
with two frigates. This is because each additional agent and its interactions with the 
others will require considerably larger computational effort. 

On a personal computer it would take approximately 250-300 days to complete 
this experiment and get the data. Fortunately, the Naval Postgraduate School’s 
Simulation Experiments & Efficient Design (SEED) Center offers a great computational 
resource for thesis students. The SEED Center can use over a hundred processors in 
parallel to make MANA runs. With this advantage, all of the runs were completed and 
the data was synthesized into a single comma-separated (CSV) file in just a few days. 

2. Analysis Tool 

The analysis tool used in this study is JMP, a statistical analysis tool developed by 
the JMP business unit of SAS Institute. It is very useful to the analyst for investigating 
and exploring the data. This software is used to interpret the data by performing analyses 
and creating graphs, data tables, charts, and reports. JMP automatically displays 
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statistical text as well as graphs and charts; this makes it a user-friendly analysis tool. The 
edition utilized in this study is JMP Pro 12. 

3. Measure of Effectiveness 

In this study, mission success is considered classifying the submarine before it 
enters the TDZ around the HVU. Two measures can be defined to represent this goal: the 
proportion of successful classification and the time to classify the submarine. The first 
MOE is the success rate, which represents the proportion of classification. The result of 
each run is a binary output: 1 or 0. “1” means that the submarine is classified and “0” 
means that the underwater contact isn’t detected or isn’t classified as a submarine before 
the submarine enters the TDZ. The average of the binary data gives us the overall 
proportion of Is in the output for each design point. If an ASW asset classifies the 
submarine, we assume this reduces the risk for the HVU. We also assess the effectiveness 
of each scenario by quantifying the time to classify the submarine. This measure is 
defined as our second MOE. That is, the earlier a submarine is detected, the better it is for 
the defenders. In some cases, early detection and classification is critical for decision¬ 
makers since it plays an important role in keeping the HVU out of harm’s way. 

4. A Quick Comparison of the Scenarios 

Two MOEs are used for evaluating the effectiveness of deployment tactics. The 
first MOE is the proportion of mission success, which represents the overall probability 
of classification before the submarine reaches the TDZ. The second MOE is the time to 
classify the submarine. 

A single CSV file with 390,000 rows of raw data is imported into the analysis tool 
JMP. First, we create a summary data table with all the raw data by averaging the MOE1 
(mission success) column for each scenario. For the Baseline Scenario, the mean of 
success in the overall replications is around 0.383. The success rate in the other scenarios 
differs due to the number and variety of the platforms. Scenario Six gives the highest 
success rate, since more platforms are employed than in the other scenarios. Table 9 
shows the success rate based on different scenarios. 
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Table 9. The proportion of successful classification in the overall 

replications. 


Scenario 

Name 

Number of 
Replication 

Number of 
Success 

Success 

Rate 

Baseline Scenario 

65000 

24953 

0.383892308 

Scenario Two 

65000 

24578 

0.378123077 

Scenario Three 

65000 

30752 

0.473107692 

Scenario Four 

65000 

29108 

0.447815385 

Scenario Five 

65000 

28664 

0.440984615 

Scenario Six 

65000 

34947 

0.537646154 


The scenarios are grouped to create different datasets. Because the factors and 
their ranges are identical for all scenarios, we can directly compare them. From this 
comparison, we can determine how a change in the configuration of ASW assets affects 
the MOE. Side-by-side box plots are particularly useful when comparing different 
datasets. They provide us a quick comparison of the scenarios. This comparison can help 
a tactical commanders choose an appropriate configuration of ASW assets. For each 
scenario, a box is created extending from the 25th percentile to the 75th percentile. The 
50th percentile is the median, which is drawn inside the box. Whiskers are the lines that 
limit a subset of the data, outside the box. 

We created two different side-by-side box plots: the proportion of successful 
classification versus scenario and time steps to classification versus scenario. Figure 21 
displays a comparison of the average mission success versus scenario. The box plot for 
Scenario Six gives higher results on the average mission success scale than the others. 
While the Baseline Scenario and Scenario Two look similar to each other, Scenario Four 
and Scenario Five also look similar. 
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Figure 21. Comparative boxplots: Mean(success) vs. scenario. 

We basically see that while assuming the helicopter and USV have the same 
sensor type and capability, the classification effectiveness of the ASW screen will be 
approximately the same. So, we turn to the second MOE, time to classify, to see if there 
is a difference. We take a subset of data for every level of the Mission Success column. 
This created two different data tables: Mission Success=l and Mission Success=0. 
Between them, the Mission Success=l data table is the one that we will use to quantify 
the second MOE. The Steps column gives the number of time steps in a scenario until the 
submarine is classified. The distributions of the “Steps” column by scenarios are 
provided in Appendix B. Figure 22 displays a comparison of the average steps versus 
scenario. When we look at the side-by-side box plot, firstly, we realize that the box plots 
of Scenario Three and Scenario Six resemble each other and that the average time steps 
are significantly less than in the other scenarios. Employing the helicopters and USVs 
together in an ASW screen formation will give us an early detection and classification 
capability. The early detection and classification of the submarine is a crucial factor in 
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ASW because they enable the task force commander to easily keep the HVU outside of 
the danger zone of the enemy submarine. 


Mean(Steps) vs. Scenario 



Scenario 


Figure 22. Comparative boxplots: Mean(steps) vs. scenario. 


5. One-way Analysis of the Means by Scenarios 

A /-test is used to examine the difference between two means and assumes that 
the samples are randomly drawn from normal populations; though the test is robust to 
nonnormality). In this study, the six scenarios are independent; therefore, another way of 
comparing the scenario means is by using a /-test. In this test, we use a significance level 
of a < 0.05. 


a. The Proportion of Successful Classification 

The scenarios were built to explore how different combinations of frigates, 
helicopters, and USVs contribute to the detection and classification of the submarine. The 
proportion of classification is the first measure for the comparison of the scenarios. We 
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use JMP to perform multiple pairwise comparisons of group means. Figure 23 shows the 
visual comparison of the scenario means in terms of the proportion of classification. 
Interpreting the comparison circles is a basic way to compare group means. If the 
comparison circles for different scenarios do not intersect or intersect slightly, the means 
of the scenarios are statistically significantly different. If the comparison circles for 
different scenarios intersect or intersect by an angle of higher than 90 degrees, the means 
of the scenarios are not significantly different. From Figure 23, we can interpret that the 
Baseline Scenario and Scenario Two are not significantly different. Scenario Six (with 
three of all the assets) is the only one that is significantly different than all the other 
scenarios. 


- ▼ Oneway Analysis of Mean(Mission Success) By Scenario 



Scenario 


Figure 23. The visual comparison of the scenario means in terms of the 

proportion of classification. 

From the detailed results, as shown in Figure 24, we can see that there are four 
comparisons among all pairwise comparisons where no statistically significant difference 
is found: 


• Baseline Scenario where two frigates and two helicopters are employed 
and Scenario Two where two frigates and two USVs are employed. 
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• Scenario Four where three frigates and three helicopters are employed 
and Scenario Five where three frigates and three US Vs are employed. 

• Scenario Three where two frigates, two helicopters, and two US Vs are 
employed and Scenario Four where three frigates and three helicopters 
are employed. 

• Scenario Three where two frigates, two helicopters, and two US Vs are 
employed and Scenario Five where three frigates and three US Vs are 
employed. 

The other pairwise comparisons show that there is a statistically significant 
difference. A detailed report that compares each pair is provided in Appendix C. 


4 Means Comparisons 

A - Comparisons for each pair using Student's t 
Confidence Quantile 
t Alpha 

i. 966 i 6 —► Significance Level 

/J LSD Threshold Matrix 

Abs(Dif)-LSD 


Scenario_6 Scenario_3 Scenario_4 Scenario_5 Basehne_$cenerio Scenario.2 


Scenario_6 

•0.03473 

0.02981 

0.05510 

0.06193 

0.11902 

0.12479 

Scenario.3 

0.02981 

•0.03473 

-0.00944 

-0.00261 

0.05449 

0.06025 

Scenario.4 

0.05510 

-0.00944 

-0.03473 

-0.02790 

0.02919 

0.03496 

Scenano_5 

0.06193 

-0.00261 

-0.02790 

-0.03473 

0.02236 

0.02813 

Baseline_Scenario 

0.11902 

0.05449 

0.02919 

0.02236 

-0.03473 

-0.02896 

Scenario.2 

0.12479 

0.06025 

0.03496 

0.02813 

02090 

-0.03473 


In LSD Threshold 
Matrix, positive 
values indicate that 
there is statistically 
significance 
between pairs 


Positive values show pairs of means that 


Connecting Letters Report 

Level 


$cenario 6 A 

0.53764615 

Scenario 3 B 

0.4*310769 

Scenario_4 B 

0.44 781538 


0.44098462 

Baseline.Scenario C 

0.38389231 

Scenario 2-C- 

0.37812308 


’ significantly different. 

The group means that are 
not sharing the same 
letter are considered as 
significantly different 


levels not connected by same letter are significantly different 


Ordered Differences Report 


Level 

- Level 

Difference 

Std Err Dif 

Lower CL 

Upper CL 

p Value 

Scenario_6 

Scenaho_2 

0.1595231 

0.0176639 

0.124793 

0.1942532 


Scenario_6 

Baseline.Scenario 

0.1537538 

0.0176639 

0.119024 

0.1884840 


Scenario_6 

$cenerio_5 

0.0966615 

0.0176639 

0.061931 

0.1313917 


Scenario_3 

Scenario_2 

0.0949846 

0.0176639 

0.060254 

0.1297148 


Scenario_6 

Scenario_4 

0.0898308 

0.0176639 

0.055101 

0.1245609 


Scenario_3 

Baseline_Scenano 

0.0892154 

0.0176639 

0.054485 

0.1239455 


Scenario_4 

Scenario_2 

0.0696923 

0.0176639 

0.034962 

0.1044225 


Scenario_6 

Scenario_3 

0.0645385 

0.0176639 

0.029808 

0.0992686 


Scenario_4 

Baseline_Scenario 

0.0639231 

0.0176639 

0.029193 

0.0986532 


Scenano_5 

Scenario_2 

0.0628615 

0.0176639 

0.028131 

0.0975917 


Scenario_5 

Baseline.Scenario 

0.0570923 

0.0176639 

0.022362 

0.0918225 


Scenario_3 

Scenario_5 

0.0321231 

0.0176639 

-0.002607 

0.0668532 

0.0698 

Scenario_3 

Scenario_4 

0.0252923 

0.0176639 

-0.009438 

0.0600225 

0.1530 

Scenario_4 

Scenario_5 

0.0068308 

0.0176639 

-0.027899 

0.0415609 

0.6992 

Baseline_Scenario Scenano_2 

0.0057692 

0.0176639 

-0.028961 

0.0404994 

0.7441 



Figure 24. Comparison of each pair for the proportion of successful 
classification using Student’s f-test. 
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The Connecting Letters Report is the simple way to analyze the differences 
between group means. The highest group mean is always shown on the top. Scenario Six 
has the highest group mean amongst the other scenarios—this is where the largest 
number of sensors are employed in the ASW screen formation. Scenario Three, Scenario 
Four, and Scenario Five form the first group, and Baseline Scenario and Scenario Two 
form the second group that share the same letter in the report. From these results, we can 
say that the scenarios that have the same number of sensors are not considered as 
significantly different. 

b. Time to Classify the Submarine 

Figure 25 shows the visual comparison of the scenario means for the time to 
classify measure of effectiveness. By looking at the comparison circles, one can see that 
Baseline Scenario and Scenario Four, Scenario Two and Scenario Five, and Scenario 
Three and Scenario Six do not display a significantly different group means. If time is an 
important factor on the mission, the ASW screen planning method should be considered 
an important factor. 



Figure 25. The visual comparison of the scenario means in terms of the time to 

classify. 
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From the detailed results, as shown in Figure 26, one can say that there are three 
comparisons among all pairwise comparisons where no statistically significant difference 
is found: 

• Baseline Scenario and Scenario Four, where the frigates are employed in 
the inner screen and the helicopters in the outer screen. 

• Scenario Five and Scenario Two, where the frigates are employed in the 
inner screen and the USVs in the intennediate screen. 

• Scenario Three and Scenario Six, where the frigates are employed in 
inner screen, the USVs in the intermediate screen, and the helicopters in 
the outer screen. 


Means Comparisons 

A - Comparisons for each pair using Student's t 
a Confidence Quantile 

t Alpha 

1.96616 fiosy -*• Significance Level 

£ LSD Threshold Matrix 

Abs(Dif)*LSD 


Scenario_5 Scenario_2 Baseline.Scenario Scenario_4 Scenario_3 Scenario_6 


Scenario_5 

-3184 

-2902 

3522 

4100 

24161 

25538 

Scenario_2 

-2902 

•3184 

324.0 

3818 

23878 

25258 

Baselme.Scenano 

3522 

3240 

-3184 

-2606 

17455 

18832 

Scenarios* 

4100 

38L8 

-260.6 

-3184 

1687.7 

18254 

Scenario.3 

24101 

2387.9 

17455 

1687.7 

•3184 

-1807 

$cenario_6 

2553.8 

2525.6 

18832 

18254 

-1807 

-3184 


In LSD Threshold 
Matrix, positive 
values indicate that 
there is statistically 
significance 
between pairs 


Positive values show pairs of means that are significantly different 

Connecting Letters Report 

The group means that are 
not sharing the same 
letter are considered as 
significantly different 

Levels not connected by same letter are significantly different 

Ordered Differences Report 


Level Mean 


Scenano 5 A 

165J1IW4 

Scenario 2 A 

16492912 

Baseline.Scenario B 

15850521 

Scenano 4 B 

15792716 

Scenario_3 C 

13786806 

Scenano 6 _ C 

13648929 


Level 

- Level 

Difference 

Std Err Dif 

Lower CL 

Upper CL 

p-Value 

Scenario_5 

Scenario_6 

2872144 

161.9253 

2553.77 

3190516 

<0001* 

Scenario_2 

Scenario_6 

2843.983 

161.9253 

252581 

3162354 

<.0001* 

Scenario_5 

Scenario_3 

2734468 

161.9253 

241610 

3052839 

<0001* 

$cenario_2 

Scenario_3 

2706306 

161.9253 

238784 

3024878 

<0001* 

Baseline.Scenario Scenario_6 

2201592 

161.9253 

188322 

2519863 

<0001* 

Scenario_4 

Scenario_6 

2143.787 

161.9253 

182542 

2462158 

<.0001* 

Baseline.Scenario Scenaho_3 

2063.915 

161.9253 

174554 

2382286 

<0001* 

Scenario_4 

Scenaho_3 

2006110 

161.9253 

1687.74 

2324482 

<0001* 

Scenario_5 

Scenario^ 

728357 

161.9253 

40989 

1046.728 

<0001* 

Scenario_2 

Scenario_4 

700196 

161.9253 

38182 

1018.567 

<0001* 

Scenario_5 

Baselme.Scenario 

670553 

161.9253 

35218 

988824 

<0001* 

$cenario_2 

Baselme.Scenario 

642391 

161.9253 

324.02 

960.763 

<0001* 

Scenario_3 

Scenario_6 

137.677 

161.9253 

-180.69 

456848 

0.3957 

Baseline.Scenario Scenario_4 

57.805 

161.9253 

-26057 

376176 

0.7213 

Scenaho_5 

Scenario_2 

28.161 

161.9253 

-29021 

346532 

0.8620 



Figure 26. Comparison of each pair for time to classify using Student’s /-test. 
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The other pairwise comparisons show that there is a statistically significant 
difference. A detailed report that compares each pair is provided in Appendix D. 

6. Regression Analysis 

Regression analysis is used in simulation analysis for quantifying the relationships 
among variables. Multiple linear regression is used in this analysis. 

a. Multiple Linear Regression 

Multiple linear regression is used to model the relationship between two or more 
explanatory variables and a response variable by fitting a linear equation to observed 
data. The mean response is modeled as a function of multiple variables. A multiple linear 
regression with p explanatory variables has an equation of the fonny = /? 0 + Pi x i + 
/? 2 x 2 + ... + PpX p , where x lt x 2 ... x p are the explanatory variables and y is the response 
variable. 

MANA uses a random number generator to randomize many properties in the 
scenarios. Therefore, the scenarios can produce different results for any design point each 
time they are run. The mission success is the response variable to conduct a multiple 
linear regression. Since the response variable has two levels, it is hard to fit a linear 
regression model. Another way to fit a linear regression is to summarize the data by 
calculating the means of each input combination. Therefore, a probability of mission 
success is produced for each design point. This new data table consists of 390 rows and a 
new response variable named Mean(Mission Success), which is a continuous variable that 
ranges from zero to one. By fitting a linear regression model, we can predict the 
probability that the response is equal to one (success). 

In this new table, the data points for Scenario Two and Scenario Four are 
excluded. We assume that the frigates and helicopters have already employed in all the 
scenarios. The number of frigates and USV presence are considered as categorical factors 
to explore their effectiveness on the response. 


60 



We examine the distribution of the mean of the response using the distribution 
platform in JMP. Figure 27 shows that the mean of the response is approximately 
normally distributed with a mean of 0.461 and standard deviation of 0.114. 
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Figure 27. Distribution for the mean response. 


b. Main Effects Model 

In order to understand the relationship between the input factors and response, a 
model is fitted using only the main factors without any interactions. We look at the actual 
by predicted plot to evaluate the goodness of fit of the model. From Figure 28, the actual 
by predicted plot shows that the model fits the data quite well. In this model, the R- 
squared value is around 0.91. The R-squared value is a statistical measure which 
represents how well the regression line approximates the data points. This model explains 
91% of the variance of the data. 
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^ Actual by Predicted Plot 

0.8 





0.2 0.3 0.4 0.5 0.6 0.7 0.8 

Mean(Mission Success) Predicted P<.0001 
RSq=0.91 RMSE=0.0363 


1 Summary of Fit_| 

RSquare 

0.905435 

RSquare Adj 

0.898372 

Root Mean Square Error 

0.036285 

Mean of Response 

0.460615 

Observations (or Sum Wgts) 

260 


Figure 28. Actual by predicted plot and the summary of the fit for the main 

effects model. 


Is this main effect model the correct model to capture the relationship between the 
input factors and response variable? To answer this question, we examined four different 
assumptions related to the residuals. A residual value represents the distance between the 
observed value and the fitted value in the model. A graphical representation is an 
effective way to evaluate the adequacy of the model. Figure 29 displays the distribution 
of the residuals with graphs, quantiles, and summary statistics. From Figure 29, we see 
two assumptions are satisfied: the residuals are approximately normally distributed, and 
the mean of the residuals is approximately equal to zero. A normal Q-Q plot is also used 
to assess the normality. We can see that the approximate linearity of the points on this 
plot indicates that the residuals are normally distributed. 
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Distributions 

Residual Mean(Mission Success) 



Quantiles 

100.0% maximum 
99.5% 

97.5% 

90.0% 

75.0% 

50.0% 

25.0% 

10 . 0 % 

2.5% 

0.5% 

0 . 0 % 


Summary Statistics 


quartile 

median 

quartite 


0.1166082446 CM«n 
0.1113140012 Std Dev 
0.0731903673H 
0.048864244 
01 229480228 
-0.00396694' 

-0 : 2 3 803345 
-0.044405603 
-0.062569617 
-0.089485071 
-0.09 91-5943 


The approximate 
linearity of the points 
indicates that the 
residuals are normally 
distributed. 


The residuals are 
approximately 
normally distributed 


Std Em Mean 
Upper 95% Me n 
Lower 95% Me#n 
N 


5.5Sle-17ES 


0.0350013 

0.0021707 

0.0042744 

-0.004274 

260 


The mean of the 
residuals are 
equal to zero 


-0.12 -0.09 -0.06 -0.03 0 0.031 0.062 0.093 0.124 


Figure 29. Distribution of the residuals for the main effects model. 


For fitting a valid model, the error terms must be uncorrelated and have constant 
variance. To check these assumptions, we created the residual by predicted plot shown in 
Figure 30. We see that the residuals are scattered randomly about zero and they have 
constant variance. The assumption of uncorrelated errors is also satisfied because there is 
no evidence of sequencing of points. 
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Residual by Predicted Plot 
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Figure 30. Residual by predicted plot for the main effects model. 


The sorted parameter estimates report is useful in screening situations. This report 
shows the estimates of the parameters and conducts a hypothesis test for each model 
parameter to test the claim that the parameter estimate is equal to zero. In Figure 31, the 
parameter estimates are sorted according to their significance level. The most significant 
effects can be seen at the top of the report. There are 13 highly significant factors, one 
significant factor, and four insignificant factors. The most statistically significant factor is 
Frigate Sonar Classification Range, which represents the reality in an ASW screen 
formation. USV Presence is the second one that highly affects the response. All USV- 
related factors are marked in Figure 31. Among these factors, USV Speed is the only 
statistically insignificant factor in the model. The value of each estimate has a direct 
interpretation on the response. For example, the presence of a USV increases the 
probability of mission success by 0.089. 
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Sorted Parameter Estimates 


Term 

Frigate Sonar Classification Range 

lUSVPresencefl-Ol I 

Helicepter Sonar Classification Range 
Frigate Classification Probability 
Submarine Personel Concealment per Detection 
*Frigate[3-2] 

Submarine Attack Speed 

|uSVSonarClassification_Ran2e^^^_^^^_] 

Helicopter Average Time Between Detections 
Helicopter Dipping Time 

Helico£te^Classjfication ii Probabilit^^^_^^^^ 

USV Average Time Between Detections 
USV Sonar Classification Probability 
USV Dipping Time 

Helicopter Speed 

] 

Helicopter Refuel Time 


■HditBBtsf Endurance 

lusv Speed 


Estimate Std Error t Ratio 


0.0000501 

2.099e-6 

0.0893869 

0.004501 

0.0000257 

1.4e-6 

0.6106949 

0.038383 

-0.005531 

0.000384 

0.0642308 

0.004501 

-0.015877 

0.001537 

0.0000138 

1.4e-6 

-0.000296 

4.266e-5 

0.0001482 

2.56e-5 

0.1652491 

0.038411 

-0.00014 

4.266e-5 

0.1014717 

0.038395 

0.0000563 

2.561e-5 

0.000203 

0.000192 

1.4141e-6 

2.133e-6 

0.0003367 

0.000509 

-4.184 e-6 

6.399e6 



Prob> |t| 

<.0001* 

< 0001 * 

<■ 0001 * 

<.0001* 

< 0001 * 

<. 0001 * 

<•0001* 

<. 0001 * 

<.0001* 

<.0001* 

<.0001* 

0 . 0012 - 

0.0088- 

0.0289* 

0.2916 

0.5080 

0.5093 

0.5138 


Figure 31. The sorted parameter estimates for the main effects model. 


JMP produces a prediction expression which shows the equation used to predict 
the response (see Figure 32). This expression can be very useful in the decision-making 
process. 


Prediction Expression 

-0.1629833231228 

2 =>0 

♦ Matdi( sFngate) 3 = 0.06423076923077 

else 3 . 

0 =0 

♦ Match] USVPresence J 1 3 0.08938693746909 

else 3 . 

♦ 0.00020299882666-HelicopterSpeed 

♦ 0.00033669543304 • USV Speed 


+ -0.000296496107- Helicopter Average TimeBetween Detections 
+ -0.000139538825* USVAverageTimeBetween Detections 

♦ 0.00000141407401" Helicopter Endurance 

♦ -0.0000041844127* Helicopter Refuel Time 

+ 0.00014816251092* Helicopter Dipping Time 
+ 0.0000563050785* USV Dipping Time 

♦ -0.015876524082* SubmarineAttackSpeed 

♦ -0.0055306653183 * Submanne Personel Concealmentper Detection 

♦ 0.00005010821762* Frigate Sonar Classification Range 

♦ 0.00002570762924* HelicopterSonar Classification Range 
+ 0.00001379568173* USVSonarClassification Range 

♦ 0.61069488533826* Frigate Classification Probability 

♦ 0.16524913885334 * HelicopterClassification Probability 

♦ 0.10147174064001* USVSonarClassification Probability 


Figure 32. Prediction expression for the main effects model. 
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c. Second Order Model 

In this section, a second order regression model is developed over the data set. We 
included main effects, two-way interactions, and second-order polynomial terms in the 
model. A stepwise regression technique is used to select a subset of effects to fit a better 
model. When the additional terms are added to the model, the R-squared value will 
increase. It is good to have a higher R-squared value, but it is also desirable to have fewer 
tenns in the model to avoid overfitting the data. In brief, we are trying to fit a valid 
parsimonious model. Therefore, we created a table using the stepwise regression step 
history report and then, we plotted R-squared vs. the number of terms. As Figure 33 
suggests, after the 23rd term, the R-squared value reaches a point where adding more 
terms will not improve our model much. 

R-Squared vs. Number of Terms _ • RSquare 

|-Smooth 


05 


05 


■ 


05 


04 


03 

0 10 20 30 40 50 

Number of Terms 

Figure 33. R-squared value increases with the added terms. 


R-Squared vs. Number of Terms 
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The actual by predicted plot and summary of fit for this model is shown in Figure 
34. The second order model’s predictions seem very good. The R-squared value is around 
0.96. The second order model explains 96% of the variance of the output. 


a Actual by Predicted Plot 

0.8 


3 

c 

< 


0 ) 

5 



0.1 02 03 04 0.5 0.6 0.7 0.8 

Mean(Mission Success) Predicted P<.0001 
RSq=0.96 RMSE=0.0242 


J Summary of Fit 

RSquare 

0.958707 

RSquare Adj 

0.954874 

Root Mean Square Error 

0.024179 

Mean of Response 

0.460615 

Observations (or Sum Wats) 

260 


Figure 34. Actual by predicted plot and the summary of the fit for the second 

order model. 


The R-squared value is really high, but it does not fully guarantee that the second 
order model fits the data well. We need to check the residual distributions to investigate 
how well this model fits the data. Figure 35 displays the distribution of the residuals with 
graphs, quantiles, and summary statistics. This figure confirms that the residuals are 
distributed around zero and follow a normal distribution. For smaller samples, JMP 
provides the Shapiro-Wilks test, which tests whether the data comes from a normal 
distribution. Because the p-value is greater than .05, we retain the null hypothesis that the 
residuals come from a normally distributed population. 
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0.0644849866 < 
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0.0492108589 
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00231289 
0.0014344 
m 0.0028246 
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The mean of the 
residuals are 
equal to zero 


a » Fitted Normal 
A Parameter Estimates 

Type Parameter Estimate Lower 95% Upper 95% 


Location p 4.548e-17 -0.002825 

Dispersion o 0.0231289 0.0212971 

-2log(Likelihood) = -1221.82175254061 

A Goodness-of-Fit Test 

Shapiro-Wilk W Test 

W Probe W 

0.994942 CO. 547? 


0.0028246 

0.0253082 


Retain the null hypothesis 
the residuals are normally 
distributed 


Note: Mo - The data is from the Normal distribution. Small p- 
values reject Ho. 


Figure 35. Distribution of the residuals for the second order model. 


The residual by predicted plot in Figure 36 indicates that the residuals have 
constant variance and follow a random pattern. Therefore, the second order model 
satisfies the assumptions well. 


Residual by Predicted Plot 
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Mean(Mission Success) Predicted 


Figure 36. Residual by predicted plot for the second order model. 
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The sorted parameter estimates are shown in Figure 37. There are 15 highly 
significant factors, four significant factors, and four insignificant factors. The most 
statistically significant factor is USV Presence in the second order model, while Frigate 
Sonar Classification Range is the most statistically significant in the main effects model. 
USV Speed factor is not included in the second order model as distinct from the main 
effects model. 


Sorted Parameter Estimates 
Term _ 

kjSVPfWCTCCll-01 


Helicopter Sonar Classification Range 
Fngate Classification Probability 
Fngate Sonar Classification Range 
Submanne Personal Concealment per Detection 
*Fngate{3-2] 

^uLmanng Allaf It Sperd 




Helicopter Average Time Between Detections 
Helicopter Dipping Time 
Helicopter Classification Probability 


|USVPfM<ncg[l-0]*(USV Avtftot Time Detectiom*270 031) 


|Htl.tocwAvtf«acT,nn«B«<v»wnDtn«.oni.;70 031)-|Htl«ociffSon t .Cl«w>K»t.on R»nst-6401 02) 

|uSY?fesence[l-Or USV Dipping Tirre-450 031) I 

*Fr»gate{3-2]’(FngateSonar ClassificationRange-7315.02) 

(Submanne Attack Speed-7.50154)*(Submahne Personel Concealment per Detecbon-50.0185) 
(Submanne Personel Concealment per Detection-S0.0185)*(USV Sonar Classification Probability-0.50006) 

U?vPresence(i.5]-lU?VSonar Classification Probabl.t>-5.5W36)- 

USV Dipping Time 
USV Sonar Classification Range 
USV Sonar Classification Probability 
USV Average Time Between Detections 


Estimate 

Std Error 

t Ratio 

0.0893428 

0.002999 

29.79 

2.5739*5 

9.327*7 

2760 

0.6106851 

0.025571 

2388 

4.5026*5 

1.978*6 

22.77 

-0.005533 

0.000256 

•2163 

00642306 

0.002999 

2142 

-0.015867 

0.001024 

-1560 

25943*5 

1 865*6 

13.91 

•OuOQ 0299 

2843*5 

•1043 

0.0001484 

0.000017 

8.70 

0.1659028 

0.025582 

6.49 

-0.00031 

5685*5 

*5.46 

-1.093*7 

2.147*8 

•5.09 

0.0001534 

3.41*5 

4.50 

1.014*5 

2.797*6 

3.63 

: 0006361 

0.00015 

3.57 

0.0149024 

: 904662 

3.21 

: 1606662 

0.051145 

3.14 

-1.951*5 

2.412*5 

-0.81 

8.1928*7 

1.319*6 

0.62 

0.0210901 

0.036174 

0.58 

1.6239*5 

4.02*5 

0.40 



Figure 37. The sorted parameter estimates for the second order model. 


7. Regression Tree 

When the model has non-linearity and lots of interactions among factors, building 
a single regression model may not be enough. An alternative technique for exploring the 
effects of the factors on the response is building a regression tree. The purpose is to fit a 
model that predicts the response variable based on design factors. In regression tree 
analysis, the data is recursively partitioned into smaller regions, where the interactions 
are easy to understand; then, a predictive model is fitted for each cell of the partition. A 
tree of decision rules is formed until the desired fit is obtained. A regression tree is a 
useful technique because an analyst can easily present the results and insights. 

We build a regression tree for the probability of successful classification given all 

design factors. The data is partitioned into two segments based on the LogWorth statistic, 

which is defined as -log 10 (p-value). This statistic is reported in node Candidate 
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reports, as shown in Figure 38. The Frigate Sonar Classification Range column has the 
largest LogWorth, which is noted by an asterisk. Therefore, this factor defines the first 
optimum split. 


All Rows 



Count 260 



Mean 0.4606154 



Std Dev 0.11382 



^ Candidates 

Term 

Candidate SS 

LogWorth Cut Point 

*Frigate 

02681634615 

5.77934102 2 

USVPresence 

0.5209347846 

11.56889989 0 

Helicopter Speed 

0.1062958815 

1.33357843 100 

USV Speed 

0.0311524159 

022346943 22 

Helicopter Average Time Between Detections 

0J.268804907 

1.72272943 228 

USV Average Time Between Detections 

0.1215830805 

1.61955524 191 

Helicopter Endurance 

0.0370256410 

023647750 7594 

Helicopter Refuel Time 

0.1131237863 

145762245 2269 

Helicopter Dipping Time 

0.1254575034 

1.69488607 567 

USV Dipping Time 

0.0806772924 

037458623 309 

Submarine Attack Speed 

0.2429445949 

4.24414142 63 

Submarine Personel Concealment per Detection 

0.2692261597 

436637589 57 

Friqate Sonar Classification Range 

0.6422193893 * 

15.54367025 6458 

Helicopter Sonar Classification Range 

0.3779089892 

733367185 8630 

USV Sonar Classification Range 

0.1902161778 

3.04329916 6915 

Frigate Classification Probability 

03654461597 

7.30213508 0.303 

Helicopter Classification Probability 

0.0520842684 

0.42995882 0.528 

USV Sonar Classification Probability 

0.0832794278 

031874820 0.428 


Figure 38. Candidates report for the root node. 


The first five splits of the regression tree are shown in Figure 39. The 
interpretation of the regression tree is straightforward. Each leaf in the decision tree 
includes the probability of successful classification in the Mean row. The first split of the 
partition tree occurs with the factor Frigate Sonar Classification Range, as stated above. 
This factor is the most significant one in the regression tree model as well as in the main 
effects regression model. The original 260 design points are split into two parts: a left leaf 
that has 68 design points and a right leaf that has 192 design points. If the frigate sonar 
classification range is less than or equal to 6,343 meters, the ASW screen has a lower 
probability of successful classification. The higher probability of successful classification 
is evident when the frigate sonar classification range is higher than or equal to 6,343 
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meters. For the left leaf, the next split would happen on the factor Frigate Sonar 
Classification Probability. For the right leaf, the next split would happen on the factor 
USV Presence which is a two-level categorical variable. When USVs are not present in 
the model, the probability of successful classification is 0.4461. When USVs are present 
in the model, the probability of successful classification increases to 0.5246. The next 
split occurs for the USV Sonar Classification Range. The probability of successful 
classification is 0.4912 when the USV sonar classification range is less than 7,001 
meters, while the probability of successful classification is 0.5769 when the USV sonar 
classification range is greater than or equal to 7,001 meters. 
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0.4606154 

15.54367 

011309 


Std Dev 
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Count 192 LogWorth Difference 

Mean 0.4901927 1019667 0.08511 

StdDev 0.1008202 
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■ 

0.40 < Mean < 0.44 

0.44 < Mean < 0.48 


0.48 < Mean < 0.52 
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_ 


- USVPresence(O) 


Count 

96 

LogWorth Difference 

Mean 

04476354 

5.5143214 089432 

Std Dev 
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l 


• USVPresence(l) 

Count 96 LogWorth Difference 

Mean 0.53275 68920139 0.0935 

StdDev 08934372 


~T 


Frigate Sonar Classification 
Range<8515 
Count 72 

Mean 0.4240556 
Std Dev 0.073273 
Candidates 


w Frigate Sonar Classification 
Range>-8515 
Count 24 

Mean 0518375 

StdDev 08984673 
Candidates 


w USV Sonar Classification 
Range<7001 
Count 58 

Mean 0.4957414 
StdDev 08775336 
I b Candidates 


w USV Sonar Classificatior 
Range> = 7001 
Count 38 

Mean 0.5892368 
StdDev 08879276 
Candidates 


Figure 39. The first five splits of the regression tree. Colors and associated 
means are explained in the legend (located at the top right). 


We performed the splitting process repeatedly to find a better R-squared value. 
But, this big tree seems complex, making it difficult to display and interpret. Finally, we 
come up with 23 splits and observe an R-squared value of 0.766. Figure 40 shows a plot 
of R-squared versus the number of splits named as split history. Figure 41 displays a 
report showing each factor’s contribution to the fit in the model. 
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d Split History 



Figure 40. Split history for the regression tree model. 


J Column Contributions 

Number 

Term of Splits SS Portion 
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Submarine Personel Concealment per Detection 

2 0.16873481 





0.0656 

Helicopter Sonar Classification Range 

2 0.07726155 






0.0301 

USV Sonar Classification Probability 

1 0.05546453 






0.0216 

Submarine Attack Speed 

1 0.0421085 






0.0164 

Helicopter Classification Probability 

1 0.02839225 






0.0110 

Helicopter Speed 

1 0.0274769 






0.0107 

Helicopter Refuel Time 

1 0.02219761 






0.0086 

Helicopter Dipping Time 

1 0.02021761 






0.0079 

USV Speed 

0 0 






0.0000 

Helicopter Average Time Between Detections 

0 0 






0.0000 

USV Average Time Between Detections 

0 0 






0.0000 

Helicopter Endurance 

0 0 






0.0000 

USV Dipping Time 

0 0 






0.0000 


Figure 41. Column contributions report shows each factor’s contribution to the 

fit in the model. 
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V. CONCLUSIONS 


A. SUMMARY 

This research explores how USVs can complement and extend existing ASW 
screen effectiveness in detecting and classifying diesel-electric submarines. When an 
HVU is screened by a task force that is conducting protective ASW operations, the 
submarine threat level can be greatly reduced with high detection and classification 
capabilities. 

In this study, the modeling first focuses on building an existing ASW screening 
scenario. This baseline scenario provides a standardized benchmark to evaluate the other 
scenarios. A generic scenario was built to increase the understandability. The scenarios 
are implemented in the simulation modeling platform MANA. We have to state that the 
scenarios built in MANA may not necessarily represent the real ASW operations and the 
assumptions we made about the detection and classification of the submarine may not be 
necessarily true. Thus, this simulation study cannot answer the detailed questions, but it 
provides some useful insights about the employment of USVs in ASW screen formation. 

B. ANSWERING RESEARCH QUESTIONS 

The following research questions were presented in Chapter 1: 

1. Can USVs give the same effectiveness as ASW helicopters against diesel- 
electric submarines ahead of naval convoys or HVUs? 

2. What are the main advantages and disadvantages of employing USVs in 
an ASW screen fonnation? 

3. Which characteristics of USVs are the most significant in ASW? 

4. How do changes in decision parameters affect the probability of 
classifying a diesel-electric submarine? 

5. What strengths and drawbacks does the simulation software MANA have 
for modelling ASW scenarios? 

To answer the first question, we conduct a comparison analysis of the scenarios 
where different numbers and varieties of platforms are employed in an ASW screen 
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formation. In protective ASW operations, employing different platform and sensor types 
can help an ASW commanders detect and classify the stealthy submarine. These 
platforms and their sensors support and complement each other to improve ASW 
effectiveness. With side-by-side box plots and one-way analysis of the means by 
scenarios, we find that when the helicopters are replaced with USVs, which have the 
same sensor type and capability, they can provide the same classification effectiveness in 
an ASW screen formation. The operating range of the USVs is considered shorter than 
the operating range of the helicopters because of the autonomy requirements of USVs. 
Therefore, USVs are employed in the intennediate screen while the helicopters are 
employed on the outer screen. This gives the helicopters a great advantage against USVs 
because the helicopters can extend the reach of the frigates to the farther point in the 
ASW screen and provide an early detection and classification of the diesel-electric 
submarine. 

Addressing the second question, we show the primary advantage of employing 
USVs in ASW screen formation is freeing the helicopters to perform other missions, such 
as intelligence, surveillance and reconnaissance (ISR). The main disadvantage of 
employing USVs is that they are not nearly as efficient as the helicopters in early 
detection and classification when an early classification of the enemy submarine is 
critical for decision makers. The other disadvantage would be that USVs require a high 
level of autonomy and onboard processing for this mission, which means a higher cost 
for the development of the dipping sonar and system design. 

The proportion of successful classification is used to measure the effectiveness of 
ASW screen formation in the regression model. Based on this MOE, the most significant 
characteristic of USVs is the classification range of dipping sonar. In ASW, the 
classification range will depend on underwater conditions, background noise in the 
ocean, and sonar capability. The sonar parameters are mostly controllable because the 
selection of the sonar type and capability can be decided on during the design process. 
But, the effectiveness of sonar is limited by environmental conditions. On the other hand, 
USV speed is viewed as an insignificant characteristic in the model. The reader must 
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realize that US Vs are self-deployed to the intermediate screen ahead of the HVU with 
sufficient time and satisfy the requirements of station-keeping. 

Many decision and noise factors have a significant effect on the response in our 
protective ASW scenario. The sonar parameters of the frigates are really significant in the 
model. The first split of the partition tree occurs with the factor frigate sonar 
classification range. This factor has the greatest effect on mission success. The number of 
frigates is another significant factor which affects the outcome. Employing one more 
frigate in the screen will affect the outcome significantly. Among the noise factors, the 
stealthiness of the diesel-electric submarine plays an important role in the model, which 
is a significant factor in littoral ASW operations. The submarine is designed to submerge 
and maneuver quietly to avoid detection. In a noisy littoral environment, the submarine 
gains extra stealth. The submarine can find shadow zones to hide from active sonar and 
approach an enemy without being detected. 

Addressing the final research question, the combat simulation platfonn MANA 
has a number of strengths to simulate maritime scenarios. It is easy to use and navigate. 
In a maritime scenario, the ships may patrol on randomly assigned routes in a box; it is 
straightforward to simulate patrol boxes and random search patterns for a specific agent. 
On the other hand, MANA has several drawbacks which need to be fixed to simulate 
maritime scenarios. First of all, to form an ASW screen effectively, the ships need to 
know and update target bearing, range, course, and speed in their situational awareness 
maps at each time step. Detailed information in an agent’s situational awareness map will 
help the agent decide on their next movements. Specific built-in naval formation types 
can be added to the next versions of MANA 

The next drawback is that, in an ASW scenario, the level of acoustic classification 
is limited to two levels: submarine and non-submarine. Therefore, it is hard to implement 
ASW contact classification procedures. In a realistic ASW scenario, there are four basic 
levels about the certainty of classification: certain submarine, probable submarine, 
possible submarine, and non-submarine. Classification procedures are important for 
deciding on ASW force tactics. 
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C. FURTHER RESEARCH 

The underwater environment and the thennal layer and their effects on the sonar 
are not simulated explicitly in the model. Since the littoral waters are complex and 
chaotic due to several reasons that are mentioned in Chapter II, underwater conditions 
can have a significant effect on the effectiveness of sonar. For future work, underwater 
conditions can be simulated to prove how these conditions effect the detection and 
classification of submarines. 

In this study, we are only interested in detecting and classifying the enemy 
submarine. The phases after the classification phase are not explicitly addressed in this 
study. USVs can contribute much more effectiveness in ASW operations. They can also 
serve as armed escorts ahead of HVUs with increased size and payload. Considering this 
fact, localization, tracking, and kill phases can be simulated in future models. 
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APPENDIX A. NOLH DESIGN SPREADSHEET 


The NOLH design spreadsheet is a useful tool for designing large-scale 
simulation experiments. In this thesis, the design points are generated by using the NOLH 
design for 16 factors. 
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APPENDIX B. DISTRIBUTIONS OF “STEPS” COLUMNS BY 

SCENARIOS 


Distributions Scenario = Basclinc Scenario Distributions Scenario=Scenario_2 
Steps Steps 


Distributions Scenario=Scenario_3 
Steps 







Quantiles 

1005% 

maximum 

32390 

99.5% 


3057754 

97.5% 


29265-15 

90.0% 


274432 

75.0% 

quartile 
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Std Dev 
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Std Err Mean 
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N 
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Distributions Scenario=Scenario_4 
Steps 
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E 
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0.0% 
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Summary Statistics 

Mean 16476555 

Std Dev 80045544 

Std Err Mean 51.060562 
Upper 95% Mean 16576537 
Lower 95% Mean 16376.773 
N 24578 

Distributions Scenario=Scenario_5 
Steps 


Quantiles 
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maximum 

32076 

99.5% 


2913247 

97.5% 


27770 

90.0% 


25078 

75.0% 

quartile 
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50.0% 

median 
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25.0% 

quartile 
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10.0% 


39663 

2.5% 


159765 

0.5% 
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0.0% 
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Mean 
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Std Dev 
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Std Err Mean 
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Upper 95% Mean 
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Lower 95% Mean 
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N 
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Distributions Scenario=Scenario_6 
Steps 






Quantiles Quantiles Quantiles 
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25.0% 

quartile 

81595 

25.0% 
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0.5% 


175718 
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Summary Statistics 
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136428 

Std Dev 
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Std Dev 
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Std Dev 
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Std Err Mean 
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Std Err Mean 
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Std Err Mean 
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Upper 95% Mean 
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Upper 95% Mean 
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Lower 95% Mean 
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APPENDIX C. DETAILED COMPARISONS REPORT FOR T-TEST 
(MOE1-THE PROPORTION OF SUCCESSFUL CLASSIFICATION) 


This detailed report provides the paired /-test comparisons of the scenarios. The 
statistical text includes the difference between the levels, standard error, and confidence 
intervals, t-ratios, p-values, and degrees of freedom. A plot is also provided for the 
comparison on the right of each report. 


zi Detailed Comparisons Report 


Comparing Scenario_2 with Baseline_Scenario 

Difference 

-0.00577 t Ratio 

-0.32661 

Std Err Dif 

0.01766 DF 

384 

Upper CL Dif 

0.02896 Prob > |t| 

0.7441 i 

Lower CL Dif 

-0.04050 Prob > t 

0.6279 A 

Confidence 

0.95 Prob < t 

0.3721 


-0.06 


J 

-0.02 0.00 0.02 0.04 0.06 


Comparing Scenario_3 with Baseline_Scenario 

Difference 

0.089215 t Ratio 

5.050708 


Std Err Dif 

0.017664 DF 

384 


Upper CL Dif 

0.123946 Prob > |t| 

<.0001* / 


Lower CL Dif 

0.054485 Prob > t 

<.0001* / 


Confidence 

0.95 Prob < t 

1.0000 

V 1 


-0.10 -0.05 0.00 0.05 0.10 


Comparing Scenario_3 with Scenario_2 

Difference 

0.094985 t Ratio 

5.377318 

Std Err Dif 

0.017664 DF 

384 

Upper CL Dif 

0.129715 Prob > |t| 

<.0001* 

Lower CL Dif 

0.060254 Prob > t 

<.0001* 

Confidence 

0.95 Prob < t 

1.0000 


-0.10 -0.05 0.00 0.05 0.10 


41 Comparing Scenario_4 with Baseline_Scenario 


Difference 0.063923 t Ratio 

Std Err Dif 0.017664 DF 

Upper CL Dif 0.098653 Prob > |t| 

Lower CL Dif 0.029193 Prob > t 

Confidence 0.95 Prob < t 

3.618846 

384 

0.0003* / 

0.0002* / 

0.9998 


-0.08 -0.04 0.00 0.04 0.08 

Comparing Scenario_4 with Scenario_2 

Difference 0.069692 t Ratio 

3.945457 


Std Err Dif 0.017664 DF 

384 / 


Upper CL Dif 0.104422 Prob > |t| 

<.0001* / 


Lower CL Dif 0.034962 Prob > t 

<.0001* / 


Confidence 0.95 Prob < t 

1.0000 
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-0.08 -0.04 0.00 0.04 0.08 












Comparing Scenario_4 with Scenario_3 

Difference 

Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 

-0.02529 t Ratio 
0.01766 DF 

0.00944 Prob > |t| 
-0.06002 Prob > t 
0.95 Prob < t 

-1.43186 

384 

0.1530 

0.9235 A 

0.0765 

\ 


-0.06 


-0.02 0.00 0.02 0.04 0.06 


Comparing Scenario_5 with Baseline_Scenario 



Difference 

Std Err Dif 
Upper CL Dif 

0.057092 

0.017664 

0.091822 

t Ratio 

DF 

Prob > |t| 

3.232139 

384 

0.0013* / 

V 


Lower CL Dif 
Confidence 

0.022362 

0.95 

Prob > t 
Prob < t 

0.0007* / 

0.9993 




t — 1 —i— 1 ——'—i—'—i— i *— 1 n 


-0.06 -0.02 0.00 0.02 0.04 0.06 


Comparing Scenario_5 with Scenario_2 

Difference 

0.062862 t Ratio 

3.55875 


Std Err Dif 

0.017664 DF 

384 / 


Upper CL Dif 

0.097592 Prob > |t| 

0.0004* / 


Lower CL Dif 

0.028131 Prob > t 

0.0002* / 


Confidence 

0.95 Prob < t 

0.9998 

VJ 


-0.08 -0.04 0.00 0.04 0.08 


Comparing Scenario_5 with Scenario_3 

Difference 

-0.03212 t Ratio 

-1.81857 

|-V 

Std Err Dif 

0.01766 DF 

384 / 


Upper CL Dif 

0.00261 Prob > |t| 

0.0698 


Lower CL Dif 

-0.06685 Prob > t 

0.9651 


Confidence 

0.95 Prob < t 

0.0349* -A 

V- 


-0.06 -0.02 0.00 0.02 0.04 0.06 


41 


Comparing Scenario_5 with Scenario_4 

-0.00683 t Ratio -0.38671 

0.01766 DF 384 

0.02790 Prob > |t| 0.6992 

-0.04156 Prob > t 0.6504 

0.95 Prob < t 0.3496 _ 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 



-0.06 -0.02 0.00 0.02 0.04 0.06 


41 Comparing Scenario_6 with Baseline_Scenario 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


0.153754 t Ratio 
0.017664 DF 
0.188484 Prob > |t| 
0.119024 Prob > t 
0.95 Prob < t 
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A Comparing Scenario_6 with Scenario_2 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


0.159523 t Ratio 
0.017664 DF 
0.194253 Prob > |t| 
0.124793 Prob > t 
0.95 Prob < t 


9.031003 

384 

<. 0001 * 

<. 0001 * 

1.0000 


- 0.20 - 0.10 0.00 0.10 0.20 


A Comparing Scenario_6 with Scenario_3 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


0.064538 t Ratio 
0.017664 DF 
0.099269 Prob > |t| 
0.029808 Prob > t 
0.95 Prob < t 



A Comparing Scenario_6 with Scenario_4 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


0.089831 t Ratio 
0.017664 DF 
0.124561 Prob > |t| 
0.055101 Prob > t 
0.95 Prob < t 



A Comparing Scenario_6 with Scenario_5 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


0.096662 t Ratio 
0.017664 DF 
0.131392 Prob > |t| 
0.061931 Prob > t 
0.95 Prob < t 
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APPENDIX D. DETAILED COMPARISONS REPORT FOR T-TEST 
(MOE2-TIME TO CLASSIFY THE SUBMARINE) 


4i Detailed Comparisons Report 


Comparing Scenario_2 with Baseline_Scenario 


Difference 

642391 t Ratio 3.967209 



Std Err Dif 

161.925 DF 384 



Upper CL Dif 

960.763 Prob > |t| <.0001* / 



Lower CL Dif 

324.020 Prob > t <.0001* / 



Confidence 

0.95 Prob < t 1.0000 

. V ,1 


-800 -400 0 200 600 


Comparing Scenario_3 with Baseline_Scenario 

Difference 

-2063.9 t Ratio 

-12.7461 

Std Err Dif 

161.9 DF 

384 

Upper CL Dif 

-1745.5 Prob > |t| 

<.0001* 

Lower CL Dif 

-2382.3 Prob > t 

1.0000 

Confidence 

0.95 Prob < t 

<.0001* 


-2000 -1000 0 1000 2000 


Comparing Scenario_3 with Scenario_2 

Difference 

-27063 t Ratio 

-16.7133 

Std Err Dif 

161.9 DF 

384 

Upper CL Dif 

-2387.9 Prob > |t| 

<.0001* 

Lower CL Dif 

-3024.7 Prob > t 

1.0000 

Confidence 

0.95 Prob < t 

<.0001* 


-3000 -1000 0 1000 2000 


A Comparing Scenario_4 with Baseline_Scenario 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


-57.80 

t Ratio 


-0.35698 

161.93 

DF 


384 

260.57 

Prob > 

|t| 

0.7213 

-376.18 

Prob > 

t 

0.6393 

0.95 

Prob < 

t 

0.3607 



-400 -200 0 200 400 


Comparing Scenario_4 with Scenario_2 

Difference 

-7002 t Ratio 

-4.32419 

Std Err Dif 

161.9 DF 

384 

Upper CL Dif 

-381.8 Prob > |t| 

<.0001* 

Lower CL Dif 

-1018.6 Prob > t 

1.0000 

Confidence 

0.95 Prob < t 

<.0001* 


-800 -400 0 200 600 
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Comparing Scenario_4 with Scenario_3 

Difference 

2006.11 t Ratio 

12.38911 


Std Err Dif 

161.93 DF 

384 

\ 

Upper CL Dif 

232448 Prob > |t| 

<.0001* / 

l 

Lower CL Dif 

1687.74 Prob > t 

<.0001* / 


Confidence 

0.95 Prob < t 

1.0000 

V 1 


-2000 -1000 0 1000 2000 


Comparing Scenario_5 with Baseline_Scenario 

Difference 

670.553 t Ratio 

4.141124 


Std Err Dif 

161.925 DF 

384 / 

\ 

\ 

Upper CL Dif 

988.924 Prob > |t| 

<.0001* / 

\ 

Lower CL Dif 

352.181 Prob > t 

<.0001* / 

\ 

\ 1 

Confidence 

0.95 Prob < t 

1.0000 

1 


-800 -400 0 200 600 


A Comparing Scenario_5 with Scenario_2 

Difference 28.16 

Std Err Dif 161.93 

Upper CL Dif 346.53 

Lower CL Dif -29021 

Confidence 0.95 

-400 -200 0 200 400 



Comparing Scenario 5 with Scenario 3 

Difference 

273447 t Ratio 

16.88722 /i 


Std Err Dif 

161.93 DF 

384 

1 

Upper CL Dif 

3052.84 Prob > |t| 

<.0001* / 


Lower CL Dif 

2416.10 Prob > t 

<.0001* / 


Confidence 

0.95 Prob < t 

1.0000 

A_ L 


-3000 -1000 0 1000 2000 


21 Comparing Scenario_5 with Scenario_4 


Difference 

Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 

728.36 t Ratio 
161.93 DF 

1046.73 Prob > |t| 
409.99 Prob > t 
0.95 Prob < t 

4.498107 

384 

<. 0001 * / 
<. 0001 * / 

1.0000 

\ 

\ 

V 1, 

i ■ 1 — 1 - 

-1000 -500 C 

1 T 1 1 

500 1000 

Comparing Scenario_6 with Baseline_Scenario 

Difference 

-2201.6 t Ratio 

-13.5963 


Std Err Dif 

161.9 DF 

384 

i 

Upper CL Dif 

-18832 Prob > |t| 

<. 0001 * / 

1 

Lower CL Dif 

-2520.0 Prob > t 

1.0000 / 

\ 

Confidence 

0.95 Prob < t 

<. 0001 * 



-2000 -1000 0 1000 2000 


86 




























4 ) 


Comparing Scenario_6 with Scenario_2 


Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


-2844.0 

t Ratio 

-17.5636 

161.9 

DF 

384 

-2525.6 

Prob > |t| 

<.0001* 

-31624 

Prob > t 

1.0000 

0.95 

Prob < t 

<.0001* 



-3000 -1000 0 1000 2000 


^ Comparing Scenario_6 with Scenario_3 

-137.68 
161.93 
180.69 
-456.05 
0.95 

-400 -200 0 200 400 


t Ratio 
DF 

Prob > |t| 
Prob > t 
Prob < t 


-0.85025 

384 

0.3957 

0.8021 

0.1979 



Difference 
Std Err Dif 
Upper CL Dif 
Lower CL Dif 
Confidence 


4J Comparing Scenario_6 with Scenario_4 


Difference 

-2143.8 t Ratio 

-13.2394 


Std Err Dif 

161.9 DF 

384 

\ 

Upper CL Dif 

-18254 Prob > |t| 

<.0001* / 

\ 

Lower CL Dif 

-246 22 Prob > t 

1.0000 / 


Confidence 

0.95 Prob < t 

<.0001* 




-2000 -1000 C 

1000 2000 

Comparing Scenario 6 with Scenario 5 

Difference 

-2872.1 t Ratio 

-17.7375 


Std Err Dif 

161.9 DF 

384 


Upper CL Dif 

-2553.8 Prob > |t| 

<.0001* I 

1 

Lower CL Dif 

-3190.5 Prob > t 

1.0000 

1 

Confidence 

0.95 Prob < t 

<.0001* 



-4000 -2000 0 1000 3000 
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